Moneyballing Movies



  • Steve Sailer is fascinated by the lessons of Moneyball (using statistics in ways that defy conventional wisdom to get superior results from an inferior budget). He's talked about it in terms of things like real estate and college admissions. His latest post on this topic is about Relativity Media, which tried to Moneyball movies. They just declared bankruptcy.

    In a 2009 Esquire article, we got some glimpse of what Relativity was doing:

    Before Relativity commits to financing a particular movie — either through its slate deals with Sony and Universal or on its own — it’s fed into an elaborate Monte Carlo simulation, a risk-assessment algorithm normally used to evaluate financial instruments based on the past performance of similar products. Enough variables are included in the Monte Carlo for Wilson and his team to have reached the limits of their Excel’s sixty-five thousand rows of data [remember, 2009]: principal actor, director, genre, budget, release date, rating, and so on. After running the movie through ten thousand combinations of variables (in marathon overnight sessions), the computers will churn out a few hundred pages that culminate in two critical numbers: the percentage of time the movie will be profitable, and the average profit for each profitable run.

    (emphasis mine)

    Sailer's response (he's not an IT focused guy, mind you) calls out TRWTF:

    Warning Flag: if you are running a billion dollar business on an Excel spreadsheet in which you’ve maxed out the capacity of Excel, you really shouldn’t be using Excel anymore. Hire yourself some professional programmers to code a program for you.

    And if the guy you are giving hundreds of millions to to invest for you keeps telling you about his giant Excel spreadsheet, offer to send some of your programmers over to help him turn it into a genuine program. See what he says.

    Nate Silver is a spreadsheet whiz who builds enormous Excel files. When they get unwieldy, he changes careers. (Here’s an amusingly bitter account by one of Silver’s former colleagues at Baseball Prospectus on the colossal spreadsheet they inherited from Silver.)

    From that last link on Silver's "Rube Goldberg Goldberg contraption of an Excel spreadsheet:"

    ...26 worksheets, 532 variables, and a 103MB file size.



  • @boomzilla said:

    Excel’s sixty-five thousand rows of data [remember, 2009]

    2 years after Excel upped that limit to over a million rows with the XLSX format.

    @boomzilla said:

    Warning Flag: if you are running a billion dollar business on an Excel spreadsheet

    And there's probably quite a few of these, and I'm sure some are too cheap to want to hire programmers to replace the spreadsheet with something more sane.



  • @ChaosTheEternal said:

    2 years after Excel upped that limit to over a million rows with the XLSX format.

    Oh come on, a new Excel license costs like >$100, that's expensive.


  • BINNED

    I wonder how they ran the simulations, if they are Excel-level competent (business executives maybe?)!
    They may have been too cheap to hire a programmer with data mining skills (or just programming
    skills for what that matters), but come on at least use R.
    That is the problem though, people who do know something, but not enough.



  • @dse said:

    They may have been too cheap to hire a programmer with data mining skills (or just programming skills for what that matters), but come on at least use R.

    If they are using Excel, what makes you think they're capable of using the usability nightmare known as R?

    That said, there are still plenty of actually-usable mathematics products out there. But they all cost more than Excel.


  • BINNED

    I personally go for Scikit-Learn, but it is considered more a software library than R which is a package for statisticians (and I myself hate its syntax/UI with passion).

    Here wakario is one higher-level product based on Scikits/Python that is still great (with nice web-based UI and all).

    EDIT:

    If you are looking for a non-web-based application and non-obvious models, Paraview is a heavy-lifting tool (very heavy in fact), it is the smartest package in existence.



  • ... whatever, the point is, you need a programmer to use R just as much as you'd need one to replace a complicated Excel sheet.


  • BINNED

    That is the point, abusing Excel to the point of having more than 100 columns, is stupid. I have seen C-level executives (and some product managers) know Excel better than most of data-mining engineers! They do all sort of crazy graphs and analysis in one big freaking spreadsheet, with highlights and all. It remembers me of the time I was using FoxPro to write a game!

    If Excel sheet is complicated it is just the time to address that by taking it out of prototype and into a product, and you are right it is a programmer's job.



  • !!!!!

    Or the real problem is that there's no good product that fills this obvious need, Excel's literally the best we got.

    In an alternative universe where IT doesn't suck ass, you could easily envision HyperCard evolving into such a thing. Or possibly RealBasic, but it was more a Access competitor.


  • BINNED

    If the product is to address the executives yes it has to avoid any sort of programming (including Basic), it should be a mix of Google Docs but with easy data visualization and analytics, that one can select a few rows then right click and do different graphs.



  • @anonymous234 said:

    Oh come on, a new Excel license costs like >$100, that's expensive.

    You didn’t think they became a successful business by spending their money on things they don’t need, did you?



  • @Gurth said:

    @anonymous234 said:
    Oh come on, a new Excel license costs like >$100, that's expensive.

    You didn’t think they became a successful business by spending their money on things they don’t need, did you?

    If a bankrupt business is successful, what exactly constitutes an unsuccessful business in your eyes?


  • BINNED

    this is called sarcasm, and is a very popular :trollface:ing tactic.



  • @Rhywden said:

    If a bankrupt business is successful, what exactly constitutes an unsuccessful business in your eyes?

    Well, clearly they went bankrupt because they spent their money frivolously on things like unnecessary software licences.



  • @Gurth said:

    @Rhywden said:
    If a bankrupt business is successful, what exactly constitutes an unsuccessful business in your eyes?

    Well, clearly they went bankrupt because they spent their money frivolously on things like unnecessary software licences.

    Indubitably.



  • @dse said:

    I wonder how they ran the simulations, if they are Excel-level competent (business executives maybe?)!
    Goal Seek. It's in the What-If Analysis plugin package.@dse said:
    it should be a mix of Google Docs but with easy data visualization and analytics, that one can select a few rows then right click and do different graphs.
    Google Sheets doesn't support the What-If Analysis tools. Building them yourself is a violation of the Google App Engine terms of service since it's considered abusive. Yet another reason to go with a thick client.


  • Grade A Premium Asshole

    @blakeyrat said:

    HyperCard

    No mention of Mac Classic. I am disappointed.


  • SockDev

    More fuel for my long-term theory that Excel is the single most damaging piece of software to computing.



  • @Arantor said:

    Excel is the single most damaging piece of software to computing.

    Perhaps. But it's probably also the single most productive piece of software, and the single most <adjective> piece of software for a large list of <adjective>s. Excel gets used a lot.


  • Discourse touched me in a no-no place

    The main problem with Excel is that there's a (soft) limit beyond which it is better to switch to a different approach, and that limit is a bit lower than where many people are using it. Some of it is because they don't think they can program (but they can produce complicated sheets with non-trivial macros, so we know that's BS), some of it might be that they see programmers as not doing what they do, and some of it might be because the route to a more maintainable program is sometimes awkward to see.

    There's nothing wrong at all with doing a first prototype of something in Excel. (Well, except for obviously wrong stuff. I wouldn't do a first prototype of a word processor in it…) Sometimes that prototype is Good Enough. Just be aware that it sometimes isn't.



  • @dkf said:

    Some of it is because they don't think they can program (but they can produce complicated sheets with non-trivial macros, so we know that's BS), some of it might be that they see programmers as not doing what they do, and some of it might be because the route to a more maintainable program is sometimes awkward to see.

    And some of it is probably sunk cost fallacy - they can't bear to see "all that time being wasted."


  • Discourse touched me in a no-no place

    @PJH said:

    they can't bear to see "all that time being wasted."

    Which is dumb because it's not wasted. They've used the time to learn what they want.



  • @dkf said:

    Which is dumb

    I never meant to imply it was at all rational....



  • @dkf said:

    There's nothing wrong at all with doing a first prototype of something in Excel. (Well, except for obviously wrong stuff. I wouldn't do a first prototype of a word processor in it…) Sometimes that prototype is Good Enough. Just be aware that it sometimes isn't.

    I suspect that's not what happened here. If they're going bankrupt with their Excel solution, then the likely reason is that their analysis is wrong. Moving that bad analysis to a more maintainable platform might do nothing but help them go bankrupt faster.

    I guess it's possible they went bust because the spreadsheet got so unmanageable that they either messed up the logic or couldn't keep up with a changing market. Given the choices Hollywood has made over the past ten years, I think the most likely problem is that they have simply lost sight of the value of a good story and forgot to factor it in.



  • If you've ever hung around on poker forums, you'll have heard them talking about their "variance." Even good, long term winning players don't win all the time. They need to compare the size of their bankroll vs their expected results.

    I suspect that these guys successfully reduced their variance (I think they started in 2003, so 12 years), but not enough for the gambling that is the movie industry. Of course, bankruptcy doesn't necessarily mean dead, and I don't think anyone here is familiar with their history to know why they got there.



  • All true. But baseball was ripe for moneyballing because the statistical analysis was already mature by the time of Billy Beane and the established management's attitudes and practices were quite different from the "optimized path". The movie industry is already driven by this type of analysis to some extent so the ability to take the old guard by surprise just isn't there. I find it hard to believe that anyone other than a hard-core stats guy like Nate Silver would even be able to tip the scales measurably their way.



  • Also there's a huge difference between investing in ball players and movies.



  • @Jaime said:

    All true. But baseball was ripe for moneyballing

    Because everything occurs in phases, and interaction is discrete. As much as baseball requires teamwork, it's the sport that requires a surprisingly low amount of it.

    Moneyballing doesn't account for one big thing in sports.

    At the end of the day, it's the last game you want to win.

    No weight was put on winning the final game, and so they lost. But you can't put weight on it, because you can't swap out your team for the last game.



  • @boomzilla said:

    the percentage of time the movie will be profitable, and the average profit for each profitable run

    This jumped out at me immediately - no mention of average loss when the movie fails. I skimmed the article to double check but didn't see anything about it there either. I really want to know if they were actually not considering that, because that would be an amazing failure for their stats wizards..


  • BINNED

    large variance needs a large buffer (deeper pockets), the problem is even if the variance is bound, it may be huge in Hollywood.



  • Good catch, though I suspect it's somewhat implicit in what's going on:

    Unless the movie shows the distinct probability of a return — no one at Relativity will reveal the precise green-light figure, but it's something like 70 percent — the script gets shredded.

    So presumably the average profit is an expected value. But yeah, you'd like to think they were looking at the range of projections. The founder is / was a hedge fund guy, so I'd expect him to be savvy enough to understand that. I wouldn't expect a journalist or his editor to, though.



  • @xaade said:

    Because everything occurs in phases, and interaction is discrete. As much as baseball requires teamwork, it's the sport that requires a surprisingly low amount of it.

    Moneyballing doesn't account for one big thing in sports.

    At the end of the day, it's the last game you want to win.

    Are you suggesting that moneyball didn't work for the 2002 Oakland A's? No method can guarantee a World Series win.



  • replying to @fwd @boomzilla

    The one "good" thing about this sort of analysis is that the downside of how much money you could lose on any one is easily quantifiable - all of it.

    See for example - this weekends box office for "The Fantastic Four" was a not-so fantastic $26 million.

    --


  • mod

    @ijij said:

    See for example - this weekends box office for "The Fantastic Four" was a not-so fantastic $26 million.

    Even just looking at numbers, The Fantastic Four remake should not have been done. Take Spider-Man as a comparison

    • The first Spider-Man movie had a budget of $139,000,000, made $403,706,375 domestic, and $821,706,375 worldwide. That's about 190% domestic ROI, and 491% worldwide ROI.
    • The second Spider-Man movie had a budget of $200,000,000, made $373,524,485 domestic, and $783,705,001 worldwide. That's about 86% domestic ROI, and 292% worldwide ROI.
    • Spider-Man 3 had a budget of about $258,000,000, made $336,530,303 domestic, and $890,875,303 worldwide. That's about 30% domestic ROI, and 245% worldwide ROI.

    So the original Spider-Man series didn't do too badly, and when they did the reboot it did ok as well:

    • The Amazing Spider-Man: budget of $220,000,000, made $262,030,663 domestic, and $757,890,267 worldwide. That's about 19% domestic ROI and 244% worldwide ROI.
    • The Amazing Spider-Man 2: budget of $200,000,000, made $202,853,933 domestic, and $708,996,336 worldwide. That's about 1% ROI domestic and 254% ROI worldwide.

    By comparison, the original Fantastic Four movies struggled:

    • First Fantastic Four: budget of $87,500,000, made $154,695,569 domestic, $330,716,569 worldwide. That's about 77 % ROI domestic and 278% ROI worldwide.
    • Second Fantastic Four (Rise of the Silver Surfer): budget of $120,000,000, made $131,921,738 domestic, $288,215,319 worldwide. That's about 10% ROI domestic and 140% ROI worldwide.

    Looking at that, the first F4 movie fits somewhere between the second and third original SPider-Man movies, and the second F4 just falls off the charts. Why would you approve a reboot of such a franchise?


  • SockDev

    @abarker said:

    Why would you approve a reboot of such a franchise?

    still made them money.

    quite a bit of money actually.

    even 10% ROI is a lot of money when you're up in the hundreds of millions of dollars investment range.


  • mod

    The reboot has not made them money. They spent $120,000,000 on it. The box office numbers so far: ~$27,600,000 worldwide.

    The value you are quoting is for Fantastic Four: Rise of the Silver Surfer.


  • SockDev

    unless i miss ijij's point that number is for a single weekend at the box office. there's still a fair bit of time to make up the rest.

    regardless even if it's a total flop there's always the insurance to cash in.


  • mod

    Here's the link where I'm getting the numbers: http://www.the-numbers.com/movies/franchise/Fantastic-Four

    Just doing basic projections, opening weekend or the current movie was less than half that of the F4 movies. That means and estimate of worldwide totals around $140 million is somewhat realistic. But there are other things to consider:

    • Reviews from critics and public alike are largely negative.
    • The day before the movie was to be released, the director (Josh Trank) tweeted the following: "A year ago I had a fantastic version of this. And it would've received great reviews. You'll probably never see it. That's reality though." The tweet was deleted within short order.[1]

    Knowing that critics and audiences don't like the movie, and that even the director doesn't like the final cut is going to keep a lot of people away. I'd be surprised if the movie breaks even.


  • SockDev

    of course naming the movie identical to the first movie in the rebooted series is not going to cause any confusion at all?

    riiiight.

    anyway, yeah with those numbers.... i hope they paid their insurance premiums on that investment

    /me is still amazed that there is such a thing as insurance on movie profitability.... especially given how cagey hollywood is about their financials.


  • mod

    @accalia said:

    of course naming the movie identical to the first movie in the rebooted series is not going to cause any confusion at all?

    riiiight.

    That probably won't be much of an issue in theaters. But once they start selling discs …

    @accalia said:

    /me is still amazed that there is such a thing as insurance on movie profitability.... especially given how cagey hollywood is about their financials.

    You aren't the only one.



  • @accalia said:

    /me is still amazed that there is such a thing as insurance on movie profitability

    I'm sure the insurance companies are moneyballing the studios. It's rare to beat insurance companies over the long term.


  • SockDev

    @abarker said:

    But once they start selling discs …

    that's at least a solved problem.

    Fantastic Four (2005)
    Fantastic Four (2015)

    well... there's still some confusion, but at least you can differentiate between them in text listings.


  • SockDev

    @Jaime said:

    It's rare to beat insurance companies over the long term.

    i used to work in the insurance industry. trust me, it's IMPOSSIBLE to beat them in the long term. even if you individually beat them, they still win on averages.



  • Well, yes, that just stands to reason, otherwise they wouldn't stay in business. It's the same logic as with those who keep trying to run up their winnings at the craps table by betting everything on each throw; no matter what you do, the odds are in their favor, not yours, and while they can afford to lose once, you cannot. Eventually, the house will take everything you put in.

    In the end, any insurance company which a) has enough customers and b) hasn't completely fallen down in producing its actuarial tables will be able to get significantly more money in than they pay out.



  • @abarker said:

    Why would you approve a reboot of such a franchise?

    I think it's a use it or lose it situation regarding the rights to the characters/property. They did the same thing before the "first" movie - they filmed and completed an entire movie on an ultra low budget, and then sat on it because they didn't actually want to release it.

    And another aspect to consider is greenlighting a movie and making it something people will pay to see are not the same thing. Sony wants a blockbuster superhero movie, but they have no idea how to do it.



  • @fwd said:

    @abarker said:
    Why would you approve a reboot of such a franchise?

    I think it's a use it or lose it situation regarding the rights to the characters/property.

    This is one of the reasons. The other was, assuming the movie succeeds (not gonna happen), to tie in with the X-Men to create a franchise to compete against the Avengers movies. (Fantastic Four is made by Fox, not Sony.)


  • mod

    @NedFodder said:

    The other was, assuming the movie succeeds (not gonna happen), to tie in with the X-Men to create a franchise

    The director a couple of the managing producers have said that they aren't sure how that would work because of the way they have set up the storylines. One of the issues that was mentioned was that there had been no mention of the F4 in the X-Men movies, but that can be explained by saying that they hadn't been given their powers yet. The big problem is that in the Fantastic Four, they are the first people with special abilities in the world, which means no mutants. That means they are in separate universes, which makes a team up movie unlikely.



  • I agree that a F4/X-Men crossover would be awkward from a storytelling point of view. I was basing my comment off of some Internet speculation I read months ago. CBA to find the original article, but here's a representative link:



  • @dkf said:

    Some of it is because they don't think they can program (but they can produce complicated sheets with non-trivial macros, so we know that's BS)

    Perhaps it's that programming seems like such a blank page. With Excel you already have the framework ready to go. And most office workers are familiar with Excel and the concept of formulas.

    My experience of monster Excel spreadsheets is that they grow by evolving. It starts with something one person created for themselves to speed up some manual process... Someone else sees it or uses it when they're on vacation... Then it gets added to and more people get access to it... Over time, without any actual decision being made, there's a monster business-critical spreadsheet that no-one fully understands.



  • @RTapeLoadingError said:

    Over time, without any actual decision being made, there's a monster business-critical spreadsheet that no-one fully understands.

    And at that point, it's by no means certain that replacing it with a monster business-critical custom application that no-one fully understands is going to be any sort of improvement.


Log in to reply
 

Looks like your connection to What the Daily WTF? was lost, please wait while we try to reconnect.