Super Crunchers (18 page)

With neural networks, the researcher just needs to feed in the raw information, and the network, by searching over the massively interconnected set of equations, will let the data pick out the best functional form. We don't have to figure out in advance how dogs' different physical attributes interact to make them better racers; we can let the neural training tell us. The Super Cruncher, under either the regression or neural methods, still needs to specify the raw inputs for the prediction. The neural method, however, allows much more fluid estimates of the nature of the impact. As the size of the datasets has increased, it has become possible to allow neural networks to estimate many, many more parameters than have traditionally been accommodated by traditional regression.

But the neural network is not a panacea. The subtle interplay of its weighting schemes is also one of its biggest drawbacks. Because a single input can influence multiple intermediate switches that in turn impact the final prediction, it often becomes impossible to figure out how an individual input is affecting the predicted outcome.

Part and parcel of not knowing the size of the individual influences is not knowing the precision of the neural weighting scheme. Remember, the regression not only tells you how much each input impacts the prediction, it also tells you how accurately it was able to estimate the impact. Thus, in the greyhound example, a regression equation might not only tell you that the dog's past win percentage should be given a weight of .47 but it would also tell you its level of confidence in that prediction: “There's a 95 percent chance that the true weight is between .35 and .59.” The neural network, in contrast, doesn't tell you the confidence intervals. So while the neural technique can yield powerful predictions, it does a poorer job of telling you why it is working or how much confidence it has in its prediction.

The multiplicity of estimated weighting parameters (which can often be three times greater with neural networks than with regression prediction) can also lead toward “overfitting” of the training data.
^*3
If you have the network “train” itself about the best 100 weights to use on 100 pieces of historical data, the network will be able to precisely predict all 100 outcomes. But exactly fitting the past doesn't guarantee that the neural weights will be good at predicting future outcomes. Indeed, the effort to exactly fit the past with a proliferation of arbitrary weights can actually hinder the ability of neural networks to predict the future. Neural Super Crunchers are now intentionally limiting the number of parameters that they estimate and the amount of time they let the network train to try to reduce this overfitting problem.

“We Shoot Turkeys”

To be honest, the neural prediction methods are sufficiently new that there's still a lot of art involved in figuring out how best to estimate neural predictions. It's not yet clear how far neural prediction will go in replacing regression prediction as the dominant methodology. It is clear, however, that there are real-world contexts in which neural predictions are at least holding their own with regression prediction as far as accuracy. There are even some cases where they outperform traditional regressions.

Neural predictions are even starting to influence Hollywood. Just as Orley Ashenfelter predicted the price of Bordeaux vintages before they had even been tasted, a lawyer named Dick Copaken has had the audacity to think that he can figure out how much a movie will gross before a single frame is even shot. Copaken is a fellow Kansas Citian who graduated from Harvard Law School and went on to a very distinguished career as a partner in the Washington office of Covington & Burling. In the past, he's crunched numbers for his legal clients. Years ago he commissioned Lou Harris to collect information on perceptions of car bumper damage. The statistical finding that most people couldn't even see small dents in their bumpers convinced the Department of Transportation to allow manufacturers to use cheaper bumpers that would sustain at most imperceptible dents.

Nowadays, Copaken is using a neural network to crunch numbers for a very different kind of client. After retiring from the practice of law, Dick Copaken founded a company that he named Epagogix (from the Aristotelian idea of inductive learning). The company has “trained” a neural network to try to predict a movie's receipts based primarily on characteristics of the script. Epagogix has been working behind the scenes because most of its clients don't want the world to know what it's doing.

But in a 2006
New Yorker
article, Malcolm Gladwell broke the story. Gladwell first learned of Epagogix when he was giving a speech to the head honchos of a major film studio. Copaken told me it was the kind of “retreat where they essentially confiscate everybody's BlackBerry and cell phone, move them to some off-campus site and for a few days they try to think the great thoughts…. And they usually invite some guru of the moment to come and speak with them and help them as they sort through their thinking. And this particular year it was Malcolm Gladwell.” Even though Gladwell's job was to tell stories to the executives, he turned the tables and asked them to tell him about some idea that was going to reshape the way films are made and viewed in the next century. “The chairman of the board started to tell him,” Copaken said, “about…this company that does these neural network projections and they are really amazingly accurate. And then, although this was all supposed to be fairly hush-hush,…the head of the studio chimed in and began to give some specifics about just how accurate we were in a test that we had done for the studio.”

The studio head was bragging about the results of a paradigm-shifting experiment in which Epagogix was asked to predict the gross revenues of nine motion pictures just based on their scripts—before the stars or the directors had even been chosen. What made the CEO so excited was that the neural equations had been able to accurately predict the profitability of six out of nine films. On a number of the films, the formula's revenue prediction was within a few million dollars of the actual gross.

Six out of nine isn't perfect, but traditionally studios are only accurate on about a third of their predictions of gross revenues. When I spoke with Copaken, he was not shy about putting dollars to this difference. “For the larger studios, if they both had the benefit of our advice and the discipline to adhere to it,” he said, “they could probably net about a billion dollars or more per studio per year.” Studios are effectively leaving a billion dollars a year on the ground.

Several studios have quickly (but quietly) glommed on to Epagogix's services. The studios are using the predictions to figure out whether it's worth spending millions of dollars to make a movie. In the old days, the phrase “to shoot a turkey” meant to make a bad picture. When someone at Epagogix says, “We shoot turkeys,” it means just the opposite. They prevent bad pictures from ever coming into existence.

Epagogix's neural equations have also let studios figure out how to improve the expected gross of a film. The formula not only tells you what to change but tells you how much more revenue the change is likely to bring in. “One time they gave us a script that just had too many production sites,” Copaken said. “The model told me the audience was going to be losing its place. By moving the action to a single city, we predicted that they would increase revenues and save on production costs.”

Epagogix is now working with an outfit that produces about three to four independent films a year with budgets in the $35–50 million range. Instead of just reacting to completed scripts, Epagogix will be helping from the get-go. “They want to work with us in a collegial, collaborative fashion,” Copaken explained, “where we will work directly with their writers…in developing the script to optimize the box office.”

But studios that want to maximize profits also have to stop paying stars so much money. One of the biggest neural surprises is that most movies would make just as much money with less established (and therefore less expensive) actors. “We do take actors and directors into account,” Copaken says. “It turns out to be a surprisingly small factor in terms of its overall weighting in the box office results.” It matters a lot
where
the movie is set. But the name of the stars or directors, not so much. “If you look at the list of the 200 all-time best-grossing movies,” Copaken says, “you will be shocked at how few of them have actors who were stars at the time those films were released.”

Old habits, like the star system, die hard. Copaken says that studio execs “still aren't listening” when it comes to cutting the ad budget or substituting lesser-named actors. The neural equation says that stars and ads often aren't worth what they cost. Copaken points out, “Nobody really knew about Harrison Ford until after
Star Wars
.”

Epagogix isn't on any kind of crusade to hurt stars. In fact, the powerful Endeavor Agency is interested in using Epagogix's services for its own clients. Copaken recently spent a morning with one of Endeavor's founders, the indomitable Ari Emmanuel. Ari Emmanuel is apparently the inspiration for the character Ari Gold, in the HBO series
Entourage
. “He drove me to a couple of major studios to meet with the head of Paramount and the head of Universal,” Copaken said. “En route, he must have fielded seventy phone calls, from everybody from Sacha Baron Cohen to Mark Wahlberg to an agent for Will Smith.” Endeavor thinks that Epagogix can not only help its clients decide whether to agree to act in a movie, but it could also help them decide whether to get their money up front or roll the dice on participating in back-end profits. In an Epagogix world, some stars may ultimately be paid less, but the savvy stars will know better how to structure their contracts.

It shouldn't surprise you, however, that many players in the industry are not receptive to the idea of neural prediction. Some studios are utterly closed-minded to the idea that statistics could help them decide whether to greenlight a project. Copaken tells the extraordinary story of bringing two hedge fund managers to meet with a studio head. “These hedge fund guys had raised billions of dollars,” Copaken explained, “and they were prepared to start with $500 million to fund films that would pass muster by our test and be optimized for box office. And obviously if the thing worked, they were prepared to expand rapidly. So there was a huge source of money on the table there.” Copaken thought that he could at least pique the studio's interest with all of that outside money.

“But the meeting was not going particularly well and there was just a lot of resistance to this new way of thinking,” Copaken said. “And finally one of these hedge fund guys sort of jumped into the discourse and said, ‘Well, let me ask you a question. If Dick's system here gets it right fifty times out of fifty times, are you telling me that you wouldn't take that into account to change the way you decide which movies to make or how to make them?' And the guy said, ‘No, that's absolutely right. We would not even if he were right fifty times out of fifty times…. [S]o what if we are leaving a billion dollars of the shareholders' money on the table; that is shareholders' money…. Whereas if we change the way we do this, we might antagonize various people. We might not be invited. Our wives wouldn't be invited to the parties. People would get pissed at us. So why mess with a good thing?'”

Copaken was completely depressed when he walked out of the meeting, but when he looked over he noticed that the hedge fund guys were grinning from ear to ear. He asked them why they were so happy. They told him, “You don't understand, Dick. We make our fortunes by identifying small imperfections in the marketplace. They are usually tiny and they are usually very fleeting and they are immediately filled by the efficiency of the marketplace. But if we can discover these things and we can throw massive resources at these opportunities, fleeting and small though they may be, we end up making lots of money before the efficiency of the marketplace closes out that opportunity. What you just showed us here in Hollywood is a ten-lane paved highway of opportunity. It's like they are committed to doing things the wrong way and there seems to be so much energy in the culture and commitment to doing it the wrong way, it creates a fantastic opportunity for us that is much more durable and enduring than anything we've ever seen.”

The very resistance of some studios creates more of an opportunity for outsiders to come in and see if optimized scripts really do sell more tickets. Epagogix itself is putting its money where its mouth is. Copaken is planning to remake a movie that was a huge commercial disappointment. With the help of the neural network, he thinks just a few simple changes to the script could generate a twenty-three fold increase in the gross. Copaken's lined up a writer and plans to commission a script to make just these changes. We may soon see whether a D.C. lawyer armed with reams of data can perform a kind of cinematographic alchemy.

The screenwriter William Goldman famously claimed that when it comes to picking movies, “Nobody,
nobody
—not now, not ever—knows the least goddamn thing about what is or isn't going to work at the box office.” And maybe nobody does. Studio execs, even after years of trial and error, have trouble putting the right weights on the components of a story. Unlike machines, they can emotionally experience a story, but this emotion is a double-edged sword. The relative success of Epagogix's equations stem, in part, from its dispassionate weighting of what works.

Why Not Now?

Teasing out the development of technology and techniques helps explain why the Super Crunching revolution didn't happen earlier. Still, we should also ask the inverse question: why are some industries taking so long to catch the wave? Why have some decisions been resistant to data-driven thinking?

Sometimes the absence of Super Crunching isn't a problem of foot-dragging or unreasonable resistance. There are loads of decisions about which there just isn't adequate historical data to do any kind of statistical test, much less a Super Crunch. Should Google buy YouTube? This kind of one-off question is not readily amenable to data-driven thinking. Super Crunching requires analysis of the results of repeated decisions. And even when there are repeated examples, it's sometimes hard to quantify success. Law schools must decide every year which applicants to admit. We have lots of information about the applicants, and tons of data about past admitted students and the careers they've gone on to have. But what does it mean to be successful after graduation? The most obvious proxy, salary, isn't a great indicator; a leader in government or public interest law might have a relatively low salary, but still make us proud. If you can't measure what you're trying to maximize, you're not going to be able to rely on data-driven decisions.