Fooled by Randomness (20 page)

Recall that the survivorship bias depends on the size of the initial population. The information that a person derived some profits in the past, just by itself, is neither meaningful nor relevant. We need to know the size of the population from which he came. In other words, without knowing how many managers out there have tried and failed, we will not be able to assess the validity of the track record. If the initial population includes ten managers, then I would give the performer half my savings without a blink. If the initial population is composed of 10,000 managers, I would ignore the results. The latter situation is generally the case; these days so many people have been drawn to the financial markets. Many college graduates are trading as a first career, failing, then going to dental school.

If, as in a fairy tale, these fictional managers materialized into real human beings, one of these could be the person I am meeting tomorrow at 11:45 a.m. Why did I select 11:45 a.m.? Because I will question him about his trading style. I need to know how he trades. I will then be able to claim that I have to rush to a lunch appointment if the manager puts too much emphasis on his track record.

LIFE IS COINCIDENTAL

Next, we look at the extensions to real life of our bias in the understanding of the distribution of coincidences.

The Mysterious Letter

You get an anonymous letter on January 2 informing you that the market will go up during the month. It proves to be true, but you disregard it owing to the well-known January effect (stocks have gone up historically during January). Then you receive another one on February 1 telling you that the market will go down. Again, it proves to be true. Then you get another letter on March 1—same story. By July you are intrigued by the prescience of the anonymous person and you are asked to invest in a special offshore fund. You pour all your savings into it. Two months later, your money is gone. You go spill your tears on your neighbor’s shoulder and he tells you that he remembers that he received two such mysterious letters. But the mailings stopped at the second letter. He recalls that the first one was correct in its prediction, the other incorrect.

What happened? The trick is as follows. The con operator pulls 10,000 names out of a phone book. He mails a bullish letter to one half of the sample, and a bearish one to the other half. The following month he selects the names of the persons to whom he mailed the letter whose prediction turned out to be right, that is, 5,000 names. The next month he does the same with the remaining 2,500 names, until the list narrows down to 500 people. Of these there will be 200 victims. An investment in a few thousand dollars’ worth of postage stamps will turn into several million.

An Interrupted Tennis Game

It is not uncommon for someone watching a tennis game on television to be bombarded by advertisements for funds that did (until that minute) outperform others by some percentage over some period. But, again, why would anybody advertise if he didn’t happen to outperform the market? There is a high probability of the investment coming to you if its success is caused entirely by randomness. This phenomenon is what economists and insurance people call adverse selection. Judging an investment that comes to you requires more stringent standards than judging an investment you seek, owing to such selection bias. For example, by going to a cohort composed of 10,000 managers, I have 2/100 chances of finding a spurious survivor. By staying home and answering my doorbell, the chance of the soliciting party being a spurious survivor is closer to 100%.

Reverse Survivors

We have so far discussed the spurious survivor—the same logic applies to the skilled person who has the odds markedly stacked in her favor, but who still ends up going to the cemetery. This effect is the exact opposite of the survivorship bias. Consider that all one needs is two bad years in the investment industry to terminate a risk-taking career and that, even with great odds in one’s favor, such an outcome is very possible. What do people do to survive? They maximize their odds of staying in the game by taking black-swan risks (like John and Carlos)—those that fare well most of the time, but incur a risk of blowing up.

The Birthday Paradox

The most intuitive way to describe the data mining problem to a nonstatistician is through what is called the birthday paradox, though it is not really a paradox, simply a perceptional oddity. If you meet someone randomly, there is a one in 365.25 chance of your sharing their birthday, and a considerably smaller one of having the exact birthday of the same year. So, sharing the same birthday would be a coincidental event that you would discuss at the dinner table. Now let us look at a situation where there are 23 people in a room. What is the chance of there being 2 people with the same birthday? About 50%. For we are not specifying which people need to share a birthday; any pair works.

It’s a Small World!

A similar misconception of probabilities arises from the random encounters one may have with relatives or friends in highly unexpected places. “It’s a small world!” is often uttered with surprise. But these are not improbable occurrences—the world is much larger than we think. It is just that we are not truly testing for the odds of having an encounter with one specific person, in a specific location at a specific time. Rather, we are simply testing for any encounter, with any person we have ever met in the past, and in any place we will visit during the period concerned. The probability of the latter is considerably higher, perhaps several thousand times the magnitude of the former.

When the statistician looks at the data
to test a given relationship,
say, to ferret out the correlation between the occurrence of a given event, like a political announcement, and stock market volatility, odds are that the results can be taken seriously. But when one throws the computer at data, looking for just about
any
relationship, it is certain that a spurious connection will emerge, such as the fate of the stock market being linked to the length of women’s skirts. And just like the birthday coincidences, it will amaze people.

Data Mining, Statistics, and Charlatanism

What is your probability of winning the New Jersey lottery twice? One in 17 trillion. Yet it happened to Evelyn Adams, whom the reader might guess should feel particularly chosen by destiny. Using the method we developed above, researchers Percy Diaconis and Frederick Mosteller estimated at 30 to 1 the probability that someone, somewhere, in a totally unspecified way, gets so lucky!

Some people carry their data mining activities into theology—after all, ancient Mediterraneans used to read potent messages in the entrails of birds. An interesting extension of data mining into biblical exegesis is provided in
The Bible Code
by Michael Drosnin. Drosnin, a former journalist (seemingly innocent of any training in statistics), aided by the works of a “mathematician,” helped “predict” the former Israeli Prime Minister Yitzhak Rabin’s assassination by deciphering a bible code. He informed Rabin, who obviously did not take it too seriously.
The Bible Code
finds statistical irregularities in the Bible; these help predict some such events. Needless to say that the book sold well enough to warrant a sequel predicting with hindsight even more such events.

The same mechanism is behind the formation of conspiracy theories. Like
The Bible Code
they can seem perfect in their logic and can cause otherwise intelligent people to fall for them. I can create a conspiracy theory by downloading hundreds of paintings from an artist or group of artists and finding a constant among all those paintings (among the hundreds of thousand of traits). I would then concoct a conspiratorial theory around a secret message shared by these paintings. This is seemingly what the author of the bestselling
The Da Vinci Code
did.

The Best Book I Have Ever Read!

My favorite time is spent in bookstores, where I aimlessly move from book to book in an attempt to make a decision as to whether to invest the time in reading it. My buying is frequently made on impulse, based on superficial but suggestive clues. Frequently, I have nothing but a book jacket as appendage to my decision making. Jackets often contain praise by someone, famous or not, or excerpts from a book review. Good praise by a famous and respected person or a well-known magazine would sway me into buying the book.

What is the problem? I tend to confuse a book review, which is supposed to be an assessment of the quality of the book, with the
best
book reviews, marred with the same survivorship biases. I mistake the distribution of the maximum of a variable with that of the variable itself. The publisher will never put on the jacket of the book anything but the best praise. Some authors go even a step beyond, taking a tepid or even unfavorable book review and selecting words in it that appear to praise the book. One such example came from one Paul Wilmott (an English financial mathematician of rare brilliance and irreverence) who managed to announce that I gave him his “first bad review,” yet used excerpts from it as praise on the book jacket (we later became friends, which allowed me to extract an endorsement from him for this book).

The first time I was fooled by this bias was upon buying, when I was sixteen,
Manhattan Transfer,
a book by John Dos Passos, the American writer, based on praise on the jacket by the French writer and “philosopher” Jean-Paul Sartre, who claimed something to the effect that Dos Passos was the greatest writer of our time. This simple remark, possibly blurted out in a state of intoxication or extreme enthusiasm, caused Dos Passos to become required reading in European intellectual circles, as Sartre’s remark was mistaken for a consensus estimate of the quality of Dos Passos rather than what it was, the best remark. (In spite of such interest in his work, Dos Passos has reverted to obscurity.)

The Backtester

A programmer helped me build a
backtester.
It is a software program connected to a database of historical prices, which allows me to check the hypothetical past performance of any trading rule of average complexity. I can just apply a mechanical trading rule, like buy NASDAQ stocks if they close more than 1.83% above their average of the previous week, and immediately get an idea of its past performance. The screen will flash my hypothetical track record associated with the trading rule. If I do not like the results, I can change the percentage to, say, 1.2%. I can also make the rule more complex. I will keep trying until I find something that works well.

What am I doing? The exact same task of looking for the survivor within the set of rules that can possibly work. I am
fitting
the rule on the data. This activity is called
data snooping.
The more I try, the more I am likely, by mere luck, to find a rule that worked on past data. A random series will always present some detectable pattern. I am convinced that there exists a tradable security in the Western world that would be 100% correlated with the changes in temperature in Ulan Bator, Mongolia.

To get technical, there are even worse extensions. An outstanding recent paper by Sullivan, Timmerman, and White goes further and considers that the rules that may be in use successfully today may be the result of a survivorship bias.

I have to decry some excesses in backtesting that I have closely witnessed in my private career. There is an excellent product designed just for that, called Omega TradeStation, that is currently on the market, in use by tens of thousands of traders. It even offers its own computer language. Beset with insomnia, the computderized day traders become night testers plowing the data for some of its properties. By dint of throwing their monkeys on typewriters, without specifying what book they want their monkey to write, they will hit upon hypothetical gold somewhere. Many of them blindly believe in it.

One of my colleagues, a man with prestigious degrees, grew to believe in such a virtual world to the point of losing all sense of reality. Whether the modicum of common sense left in him might have rapidly vanished under the mounds of simulations, or whether he might have had none to engage in such pursuit, I cannot tell. By closely watching him I learned that what natural skepticism he may have had vanished under the weight of data—for he was extremely skeptical, but in the wrong area. Ah, Hume!

A More Unsettling Extension

Historically, medicine has operated by trial and error—in other words, statistically. We know by now that there can be entirely fortuitous connections between symptoms and treatment, and that some medications succeed in medical trials for mere random reasons. I cannot claim expertise in medicine, but have been a steady reader of a segment of the medical literature over the past half decade, long enough to be concerned with the standards, as we will see in the next chapter. Medical researchers are rarely statisticians; statisticians are rarely medical researchers. Many medical researchers are not even remotely aware of this data mining bias. True, it may only play a small role, but it is certainly present. One recent medical study links cigarette smoking to a
reduction
in breast cancer, thus conflicting with all previous studies. Logic would indicate that the result may be suspicious, the result of mere coincidence.