The New Market Wizards: Conversations with America's Top Traders (15 page)

In the literature on robust statistics you find that, in most circumstances, the best strategy is not some optimized weighting scheme, but rather weighting each indicator by 1 or 0. In other words, accept or reject. If the indicator is good enough to be used at all, it’s good enough to be weighted equally with the other ones. If it can’t meet that standard, don’t bother with it.

The same principle applies to trade selection. How should you apportion your assets among different trades? Again, I would argue that the division should be equal. Either a trade is good enough to take, in which case it should be implemented at full size, or it’s not worth bothering with at all.

You talked earlier about the pitfalls in market analysis. Can you provide some other examples?

Any meaningful approach must be invariant to the choice of units. An egregious violation of this rule occurs in a certain class of bar chart techniques. Some of these are simple (45-degree angles), and some are harebrained (drawing regular pentagons on the chart), but what they all have in common is the use of angles on a bar chart. Many of the trading technique compendia, even some that claim sophistication, treat such ideas.

There’s a simple consideration that absolutely invalidates all such angles-of-certain-size methods in a single swipe: The size of an angle on a bar chart is
not
invariant to changes of scale. For instance, consider the technique of drawing a line from the low of a move at a 45-degree angle. If you do this on two charts of the same contract but with different time and price scales, say from two different services, the 45-degree lines will be different. They will subsequently intersect the price series in different places. In fact, the angle of a line joining two prices on a bar chart is not a property of the price series at all. It depends completely on what units you use for price and time and how you space them on the chart, all of which are quite arbitrary. There are good methods and bad methods, but these angle techniques are no method.

As an aside, note that trend lines that involve connecting two or more points on the price series are invariant under changes of scale and, hence, make sense in a way that lines determined by slopes do not. On differently scaled charts, a given trend line has different slopes, but it intersects the price series in precisely the same places.

The lack of intrinsic meaning of angles on a bar chart has significance even for chart-oriented traders who do not employ angles. How sharply a trend slopes on a chart is often a psychological consideration in making a trade. If you fall prey to this influence, you’re letting the chart maker’s practical and aesthetic considerations impinge on your trading. Any trend can be made to look either gentle or steep by adjusting the price scale.

This example also highlights one of the advantages of computerized trading. A computer ignores all but what it is instructed not to ignore. If you wanted your computer system to be cognizant of slope, you would have to program this feature into it. At that point, it would become abundantly clear that the slope value depends directly on the choice of units and scales for the time and price axes.

I’ve always been amazed by how many people are either oblivious to the scale-dependent nature of chart angles or unconcerned about its ramifications. My realization of the inherent arbitrariness of slope-of-line methods is precisely why I’ve never been willing to spend even five minutes on Gann angles or works by the proponents of this methodology.

What are some of the analytical pitfalls in trading system design?

There are a lot of pitfalls in designing systems. First of all, it’s very easy to make postdictive errors.

Define “postdictive.”

Using information that can be available only after the fact. Sometimes the postdictiveness is blatant—a programming error. For example, you use the closing price in a computation to decide whether to initiate a trade before the close. This kind of problem, not at all uncommon, usually betrays its presence when you generate unrealistically good performance statistics. But there are subtler kinds of postdictive errors. The highest prices in your data are followed by lower prices, nearly by definition. If you incorporated these highest prices into a trading rule, or sneaked them in via seasonal considerations, the rule would work on your data, but only postdictively.

Any other pitfalls?

One that has been mentioned a lot is the problem of overfitting. The more degrees of freedom you have, the more your system is able to fit itself to the price series.

Please define “degrees of freedom” for the nonmathematical reader.

In its clearest form, a degree of freedom is a number, a so-called parameter, that yields a different system for every allowed value. For example, a moving average system varies depending on how many days one chooses to average. This is a degree of freedom, and its allowed values are positive integers. But there can also be hidden degrees of freedom. One can have structures within the system that can take on various alternative forms. If various alternatives are tested, it gives the system another chance to conform to past idiosyncrasies in the data.

Not only is it perilous to have too many degrees of freedom in your system, there are also “bad” degrees of freedom. Suppose a certain degree of freedom in your system impinges only on a very few oversized trends in the data and otherwise does not affect how the system trades. By affixing to accidental features of the small sample of large trends, such a degree of freedom can substantially contribute to overfitting, even though the overall number of degrees of freedom is manageable.

How do you determine to what extent the performance of a system is affected by overfitting past data as opposed to capturing truths about market behavior?

The best way is to look at hundreds of examples. Add degrees of freedom to a system and see how much you can get out of them. Add bogus ones and see what you can get. I know of no substitute for experience in this matter. Try a lot of systems. Try systems that make sense to you and ones that don’t. Try systems that have very few parameters and ones that are profligate with them. After a while, you develop an intuition about the trade-off between degrees of freedom and the reliability of past performance as an indicator of future performance.

Do you have a limit to how many degrees of freedom you would put into a system?

Seven or eight is probably too many. Three or four is fine.

What is your opinion about optimization? [Optimization refers to the process of testing many variations of a system for the past and then selecting the best-performing version for actual trading.]

It’s a valid part of the mechanical trader’s repertoire, but if you don’t use methodological care in optimization, you’ll get results that are not reproducible.

How do you avoid that pitfall?

You really are caught between conflicting objectives. If you avoid optimization altogether, you’re going to end up with a system that is vastly inferior to what it could be. If you optimize too much, however, you’ll end up with a system that tells you more about the past than the future. Somehow, you have to mediate between these two extremes.

Other than the things we have already talked about, what advice do you have for people who are involved in system development?

If the performance results of the system don’t sock you in the eye, then it’s probably not worth pursuing. It has to be an outstanding result. Also, if you need delicate, assumption-laden statistical techniques to get superior performance results, then you should be very suspicious of the system’s validity.

As a general rule, be very skeptical of your results. The better a system looks, the more adamant you should be in trying to disprove it. This idea goes very much against human nature, which wants to make the historical performance of a system look as good as possible.

Karl Popper has championed the idea that all progress in knowledge results from efforts to falsify, not to confirm, our theories. Whether or not this hypothesis is true in general, it’s certainly the right attitude to bring to trading research. You have to try your best to disprove your results. You have to try to kill your little creation. Try to think of everything that could be wrong with your system, and everything that’s suspicious about it. If you challenge your system by sincerely trying to disprove it, then maybe, just maybe, it’s valid.

Do you use chart patterns in your systems?

Most things that look good on a chart—say, 98 percent—don’t work.

Why is that?

The human mind was made to create patterns. It will see patterns in random data. A turn-of-the-century statistics book put it this way: “Too fine an eye for pattern will find it anywhere.” In other words, you’re going to see more on the chart than is truly there.

Also, we don’t look at data neutrally—that is, when the human eye scans a chart, it doesn’t give all data points equal weight. Instead, it will tend to focus on certain outstanding cases, and we tend to form our opinions on the basis of these special cases. It’s human nature to pick out the stunning successes of a method and to overlook the day-in, day-out losses that grind you down to the bone. Thus, even a fairly careful perusal of the charts is prone to leave the researcher with the idea that the system is a lot better than it really is. Even if you carry it a step further by doing careful hand research, there is still a strong tendency to bias the results. In fact, this bias exists in all scientific research, which is why they have persnickety double-blind tests. Even the most honest researcher will tend to bias data toward his or her hypothesis. It can’t be helped. When I did research by hand, I took the attitude that I had to discount my results by 20 to 50 percent.

I remember one time when I was on a flight from San Francisco to New York, I had a new system idea that I was excited about and wanted to test preliminarily off the charts. The system involved using a conventional indicator (stochastics, I believe) in an unconventional way. I tried the system on several different markets, and it seemed to do terrifically. When I eventually had the system computer tested, I discovered that it actually lost money. What happened was that my alignment between the indicator on the bottom of the chart and the price on top was off by a day or so. Since the signals tended to come during periods of rapid price movement, being off by one day could mean the difference between being on the wrong side of the market for a 500-point move (say, in a market such as the S&P) instead of on the right side—a 1,000-point ($5,000 in the S&P) difference altogether. So what had actually looked like a great system proved to be totally worthless. Ever since then, I’ve been very cautious about drawing any conclusions from hand testing. I now wait until the computer results are in.

The desire to find patterns is the same human quirk that convinces people that there is validity in superstitions, or astrology, or fortune tellers. The successes are much more startling than the failures. You remember the times when the oracle really hit the nail on the head, and you tend to forget the cases in which the prediction was ambiguous or wrong.

Your comments basically seem to imply that chart reading is just laden with pitfalls and unfounded assumptions.

Yes, it is. There may be people out there who can do it, but I certainly can’t. Every pattern recognition chart trader I know makes the trades he really likes larger than the trades he doesn’t like as much. In general, that’s not a good idea. You shouldn’t be investing yourself in the individual trades at all. And it’s certainly wrong to invest yourself more in some trades than others. Also, if you think you’re creating the profitable situation by having an eye for charts, it’s very difficult not to feel excessively responsible if the trade doesn’t work.

Which, I assume, is bad.

Yes, it’s very destabilizing.

Whereas if you have a mechanical system, that’s not a problem.

That’s right. Your job is to follow the system. If the system does something that results in losses, that’s just an expected part of the system. Your judgment might be on the line over the entire performance of your system, but there’s no sense in which your judgment is on the line on any single trade.

I fully understand the psychological advantages of a mechanical approach (assuming, of course, that it’s effective), but are you also saying that you’re skeptical of chart reading as a general approach to trading?

When I have an idea based on a chart pattern, I try to reduce it to an algorithm that I can test on a computer. If a method is truly valid, you should be able to explain it to a computer. Even if you can’t define it precisely, you should still be able to concoct an algorithm that approximately describes the pattern. If your algorithm gives you an expected gain near zero—as is typically the case—then don’t delude yourself into believing that the pattern has validity that depends on some indescribable interpretation you bring to it.