Fooled by Randomness (19 page)

A GURU’S OPINION

The fund management industry is populated with gurus. Clearly, the field is randomness-laden and the guru is going to fall into a trap, particularly if he has no proper training in inference. At the time of writing, there is one such guru who developed the very unfortunate habit of writing books on the subject. Along with one of his peers, he computed the success of a “Robin Hood” policy of investing with the least successful manager in a given population of managers. It consists in switching down by taking money away from the winner and allocating it to the loser. This goes against the prevailing wisdom of investing with a winning manager and taking away money from a losing one. Doing so, their “paper strategy” (i.e., as in a Monopoly game, not executed in real life) derived considerably higher returns than if they stuck to the winning manager. Their hypothetical example seemed to them to prove that one should not stay with the best manager, as we would be inclined to do, but rather switch to the worst manager, or at least such seems to be the point they were attempting to convey.

Their analysis presents one severe hitch that any graduate student should be able to pinpoint at the first reading. Their sample only had
survivors.
They simply forgot to take into account the managers who went out of business. Such a sample includes managers that were operating during the simulation, and
are still operating today.
True, their sample included managers who did poorly, but only those managers who did poorly and recovered, without getting out of business. So it would be obvious that investing with those who fared poorly at some point but recovered (with the benefit of hindsight) would yield a positive return! Had they continued to fare poorly, they would be out of business and would not be included in the sample.

How should one conduct the proper simulation? By taking a population of managers in existence, say, five years ago and running the simulation up to today. Clearly, the attributes of those who leave the population are biased toward failure; few successful people in such a lucrative business call it quits because of extreme success. Before we turn to a more technical presentation of these issues, one mention of the much idealized buzzword of optimism. Optimism, it is said, is predictive of success. Predictive? It can also be predictive of failure. Optimistic people certainly take more risks as they are overconfident about the odds; those who win show up among the rich and famous, others fail and disappear from the analyses. Sadly.

Nine

•

IT IS EASIER TO BUY AND SELL THAN FRY AN EGG

T
his afternoon I have an appointment with my dentist (it will mostly consist in the dentist picking my brain on Brazilian bonds). I can state with a certain level of comfort that he knows something about teeth, particularly if I enter his office with a toothache and exit it with some form of relief. It will be difficult for someone who knows literally nothing about teeth to provide me with such relief, except if he is particularly lucky on that day—or has been very lucky in his life to become a dentist while not knowing anything about teeth. Looking at his diploma on the wall, I determine that the odds that he repeatedly gave correct answers to the exam questions and performed satisfactorily on a few thousand cavities before his graduation—out of plain randomness—are remarkably small.

Later in the evening, I go to Carnegie Hall. I can say little about the pianist; I even forgot her unfamiliar foreign-sounding name. All I know is that she studied in some Muscovite conservatory. But I can expect to get some music out of the piano. It will be rare to have someone who performed brilliantly enough in the past to get to Carnegie Hall and now turns out to have benefited from luck alone. The expectation of having a fraud who will bang on the piano, producing only cacophonous sounds, is indeed low enough for me to rule it out completely.

I was in London last Saturday. Saturdays in London are magical; bustling but without the mechanical industry of a weekday or the sad resignation of a Sunday. Without a wristwatch or a plan I found myself in front of my favorite carvings by Canova at the Victoria and Albert Museum. My professional bent immediately made me question whether randomness played a large role in the carving of these marble statues. The bodies were realistic reproductions of human figures, except that they were more harmonious and finely balanced than anything I have seen mother nature produce on its own (Ovid’s
materiam superabat opus
comes to mind). Could such finesse be a product of luck?

I can practically make the same statement about anyone operating in the physical world, or in a business in which the degree of randomness is low. But there is a problem in anything related to the business world. I am bothered because tomorrow, unfortunately, I have an appointment with a fund manager seeking my help, and that of my friends, in finding investors. He has what he claims is a
good track record.
All I can infer is that he has learned to buy and sell. And it is harder to fry an egg than buy and sell. Well . . . the fact that he made money in the past may have some relevance, but not terribly so. This is not to say that it is always the case; there are some instances in which one can trust a track record, but, alas, there are not too many of these. As the reader now knows, the fund manager can expect to be heckled by me during the presentation, particularly if he does not exhibit the minimum of humility and self-doubt that I would expect from someone practicing randomness. I will probably bombard him with questions that he may not be prepared to answer, blinded by his past results. I will probably lecture him that Machiavelli ascribed to luck at least a 50% role in life (the rest was cunning and bravura), and that was before the creation of modern markets.

In this chapter, I discuss some well-known counterintuitive properties of performance records and historical time series. The concept presented here is well-known for some of its variations under the names
survivorship bias, data mining, data snooping, over-fitting, regression to the mean,
etc., basically situations where the performance is exaggerated by the observer, owing to a misperception of the importance of randomness. Clearly, this concept has rather unsettling implications. It extends to more general situations where randomness may play a share, such as the choice of a medical treatment or the interpretation of coincidental events.

When I am tempted to suggest a possible future contribution of financial research to science in general, I adduce the analysis of data mining and the study of survivorship biases. These have been refined in finance but can extend to all areas of scientific investigation. Why is finance so rich a field? Because it is one of the rare areas of investigation where we have plenty of information (in the form of abundant price series), but no ability to conduct experiments as in, say, physics. This dependence on past data brings about its salient defects.

FOOLED BY NUMBERS

Placebo Investors

I have often been faced with questions of the sort: “Who do you think you are to tell me that I might have been plain lucky in my life?” Well, nobody really believes that he or she was lucky. My approach is that, with our Monte Carlo engine, we can manufacture purely random situations. We can do the exact opposite of conventional methods; in place of analyzing real people hunting for attributes we can create artificial ones with precisely known attributes. Thus we can manufacture situations that depend on pure, unadulterated luck, without the shadow of skills or whatever we have called nonluck in Table P.1. In other words, we can man-make pure nobodies to laugh at; they will be
by design
stripped of any shadow of ability (exactly like a placebo drug).

We saw in
Chapter 5
how people may survive owing to traits that momentarily fit the given structure of randomness. Here we take a far simpler situation where
we know the structure of randomness;
the first such exercise is a finessing of the old popular saying that
even a broken clock is right twice a day.
We will take it a bit further to show that statistics is a knife that cuts on both sides. Let us use the Monte Carlo generator introduced earlier and construct a population of 10,000 fictional investment managers (the generator is not terribly necessary since we can use a coin, or even do plain algebra, but it is considerably more illustrative—and fun). Assume that they each have a perfectly fair game; each one has a 50% probability of making $10,000 at the end of the year, and a 50% probability of losing $10,000. Let us introduce an additional restriction; once a manager has a single bad year, he is thrown out of the sample, good-bye and have a nice life. Thus we will operate like the legendary speculator George Soros who was said to tell his managers gathered in a room: “Half of you guys will be out by next year” (with an Eastern European accent). Like Soros, we have extremely high standards; we are looking only for managers with an unblemished record. We have no patience for low performers.

The Monte Carlo generator will toss a coin;
heads
and the manager will make $10,000 over the year,
tails
and he will lose $10,000. We run it for the first year. At the end of the year, we expect 5,000 managers to be up $10,000 each, and 5,000 to be down $10,000. Now we run the game a second year. Again, we can expect 2,500 managers to be up two years in a row; another year, 1,250; a fourth one, 625; a fifth, 313. We have now, simply in a fair game, 313 managers who made money for five years in a row. Out of pure luck.

Meanwhile if we throw one of these successful traders into the real world we would get very interesting and helpful comments on his remarkable style, his incisive mind, and the influences that helped him achieve such success. Some analysts may attribute his achievement to precise elements among his childhood experiences. His biographer will dwell on the wonderful role models provided by his parents; we would be supplied with black-and-white pictures in the middle of the book of a great mind in the making. And the following year, should he stop outperforming (recall that his odds of having a good year have stayed at 50%) they would start laying blame, finding fault with the relaxation in his work ethics, or his dissipated lifestyle. They will find something he did before when he was successful that he has subsequently stopped doing, and attribute his failure to that. The truth will be, however, that he simply ran out of luck.

Nobody Has to Be Competent

Let’s push the argument further to make it more interesting. We create a cohort that is composed exclusively of incompetent managers. We will define an incompetent manager as someone who has a negative
expected return,
the equivalent of the odds being stacked against him. We instruct the Monte Carlo generator now to draw from an urn. The urn has 100 balls, 45 black and 55 red. By drawing with replacement, the ratio of red to black balls will remain the same. If we draw a black ball, the manager will earn $10,000. If we draw a red ball, he will lose $10,000. The manager is thus expected to earn $10,000 with 45% probability, and lose $10,000 with 55%. On average, the manager will lose $1,000 each round—but only
on average.

At the end of the first year, we still expect to have 4,500 managers turning a profit (45% of them), the second, 45% of that number, 2,025. The third, 911; the fourth, 410; the fifth, 184. Let us give the surviving managers names and dress them in business suits. True, they represent less than 2% of the original cohort. But they will get attention. Nobody will mention the other 98%.What can we conclude?

The first counterintuitive point is that a population entirely composed of bad managers will produce a small amount of great track records. As a matter of fact, assuming the manager shows up unsolicited at your door, it will be practically impossible to figure out whether he is good or bad. The results would not markedly change even if the population were composed entirely of managers who are expected in the long run to lose money. Why? Because owing to volatility, some of them will make money. We can see here that volatility actually helps bad investment decisions.

The second counterintuitive point is that the
expectation of the maximum
of track records, with which we are concerned, depends more on the size of the initial sample than on the individual odds per manager. In other words, the number of managers with great track records in a given market depends far more on the number of people who started in the investment business (in place of going to dental school), rather than on their ability to produce profits. It also depends on the volatility. Why do I use the notion of expectation of the maximum? Because I am not concerned at all with the average track record. I will get to see only the
best
of the managers, not all of the managers. This means that we would see more “excellent managers” in 2006 than in 1998, provided the cohort of beginners was greater in 2001 than it was in 1993—I can safely say that it was.

Regression to the Mean

The “hot hand in basketball” is another example of misperception of random sequences: It is very likely in a large sample of players for one of them to have an inordinately lengthy lucky streak. As a matter of fact it is very unlikely that an unspecified player somewhere does not have an inordinately lengthy lucky streak. This is a manifestation of the mechanism called regression to the mean. I can explain it as follows:

Generate a long series of coin flips producing heads and tails with 50% odds each and fill up sheets of paper. If the series is long enough you may get eight heads or eight tails in a row, perhaps even ten of each. Yet you know that in spite of these wins the conditional odds of getting a head or a tail is still 50%. Imagine these heads and tails as monetary bets filling up the coffers of an individual. The deviation from the norm as seen in excess heads or excess tails is here entirely attributable to luck, in other words, to variance, not to the skills of the hypothetical player (since there is an even probability of getting either).

A result is that in real life, the larger the deviation from the norm, the larger the probability of it coming from luck rather than skills: Consider that even if one has 55% probability of heads, the odds of ten wins is still very small. This can be easily verified in stories of very prominent people in trading rapidly reverting to obscurity, like the heroes I used to watch in trading rooms. This applies to height of individuals or the size of dogs. In the latter case, consider that two average-sized parents produce a large litter. The largest dogs, if they diverge too much from the average, will tend to produce offspring of smaller size than themselves, and vice versa. This “reversion” for the large outliers is what has been observed in history and explained as regression to the mean. Note that the larger the deviation, the more important its effect.

Again, one word of warning: All deviations do not come from this effect, but a disproportionately large proportion of them do.

Ergodicity

To get more technical, I have to say that people believe that they can figure out the properties of the distribution from the sample they are witnessing. When it comes to matters that depend on the maximum, it is altogether another distribution that is being inferred, that of the best performers. We call the difference between the average of such distribution and the unconditional distribution of winners and losers the survivorship bias—here the fact that about 3% of the initial cohort discussed earlier will make money five years in a row. In addition, this example illustrates the properties of ergodicity, namely, that time will eliminate the annoying effects of randomness. Looking forward, in spite of the fact that these managers were profitable in the past five years, we expect them to break even in any future time period. They will fare no better than those of the initial cohort who failed earlier in the exercise. Ah, the long term.

A few years ago, when I told one A., a then Master-of-the-Universe type, that track records were less relevant than he thought, he found the remark so offensive that he violently flung his cigarette lighter in my direction. The episode taught me a lot. Remember that nobody accepts randomness in his own success, only his failure. His ego was pumped up as he was heading up a department of “great traders” who were then temporarily making a fortune in the markets and attributing the idea to the soundness of their business, their insights, or their intelligence. They subsequently blew up during the harsh New York winter of 1994 (it was the bond market crash that followed the surprise interest rate hike by Alan Greenspan). The interesting part is that several years later I can hardly find any of them still trading (ergodicity).