One of the hidden dangers of system development is over-optimization. As part of your system development process you will be testing your system to see how well it has performed in the past. Some system development tools will let you quickly test thousands of combinations of system parameters in a short period of time.

As you go about evaluating the profit and loss, drawdown and smoothness of the equity curve increase, you’re likely to focus on the one set of parameters that looks best. This is the first danger. It’s better to look for a cluster of superior results than for just one outstanding set of parameters. Choosing parameters that are in the middle of a good cluster of results are likely to lead to more consistent results in the future.

You will also want to make sure you have enough trades in the resulting test set so they’re statistically significant. It’s foolish to be confident in any system for which you have a track record with less than 30 trades during the chosen test period. You can have some confidence after you have more than 300 trades in the test period. This assumes you haven’t over-optimized those results.

Even 300 trades don’t guarantee a reliable system. I have seen any number of over-optimized systems that do great for over 300 trades and then lose money forever after.

One way to check your parameters is to hold back some of your historical data as a “test set.” The danger here is that if you use the test set more than once at the very end of your system development, you’re compromising the optimization. This is because your “test set” now has become part of the optimization procedure.

Still, it’s possible to over-optimize if you have a lot of parameters.

HYPOTHETICAL EXAMPLE:
It might be possible to have a winning system if, for example, the stock you buy:
• has a trade volume between 97,368 and 143,768 shares per day
• opens with a price between $6.42 and $11.76
• rises between .39% and .76% between 9:30 and 9:53 am.

and your exit strategy is to:
• wait for the stock pick to rise an additional .64%
• close the position by 11:44 am.

And did I mention that this is only on Wednesdays close to a full moon and while you’re holding your breath during the test run? While such a system COULD have 30 winning entries in a row during the hypothetical 4 year test period, would you really have confidence that this winning streak would continue?

As you can see from this hypothetical example, even though you MIGHT have gotten some great results in the past, it’s unlikely they’ll continue in the future. In many cases, you could apply some serious math and see at which point you’re in danger of over-optimizing.

However the answer to such an exercise will still be a probability. With experience, you can often develop the judgment to know if you’re in danger of having over-optimized. In general, the fewer the parameters, the less danger there is of over-optimizing.

Frequently, successful systems will only have 2 to 5 parameters. You’re also looking for a cluster of superior results rather than just one outstanding test run.

For example, you might have a system that picks stocks based on an average volume range and a price range. Right there, you already have 4 parameters. You could add an opening price move minimum, but anything beyond that could lead to over-optimization.

In the world of probabilities, you can never be certain that you’ve got a reliable system. This is because you could always be hitting an unlikely (but still possible) string of good or bad results right out of the gate. However, by not over-optimizing, you stand a better chance of having a consistently profitable system.

As a Director of Investing Systems Network, Mr. Newberry is the primary consultant on the usability and conceptual development of portfolio management software and tools to trade stocks. Investing Systems Network makes these available to retail investors in more than 70 countries.