Backtest not proof
NEW YORK – In investing, like sports, the numbers can be compelling but don’t always tell the whole story.
Athletes and stock pickers alike can hit cold streaks or lose their rhythm when conditions change, so a solid long-term record is what really burnishes a reputation. Rookie funds are at something of a disadvantage, knowing that investors like to see a history of performance, so many of these funds will calculate what their results would have been had they been around longer. But while the numbers that come from so-called backtesting can be helpful, investors should remember there is a difference between a scrimmage and a regular game.
“It isn’t the same thing as real results,” said Russel Kinnel, a mutual fund analyst at investment research provider Morningstar Inc. “I think you certainly have to keep it in mind when it is backtested,” he said of investors considering young funds.
And even if everyone agreed backtested figures carried as much weight as a log of a fund’s actual performance, investors would be wise to remember the boilerplate statement that’s as much a part of mutual fund ads as tags on new mattresses or FBI warnings on movies: Past performance doesn’t guarantee future results.
Backtesting is often used by funds that mirror an index. If a fund follows an index, the thinking goes, it’s possible to go back and plot how the fund would have traded by looking at how the index did. But in some cases, even tracing an index back through time can present problems.
Kinnel has concerns about backtesting with funds that invest in small-cap stocks, where limited supply and demand can make it harder to buy and sell.
Some poorly constructed models might even fail to add in trading costs that over the long term would depress returns.
“There’s no reason you can’t test for liquidity and trading costs and commissions. But it takes more effort,” Kinnel said. “You have to make it realistic. And that involves realistic costs.”
It’s not that funds are likely trying to use misleading data, analysts say. It can simply be difficult to account for all the variables that could have affected how a fund performed. Indeed, if a fund isn’t following an index and is instead run using a strategy it can be hard to pinpoint what move fund managers might have made.
Mark Labovitz, manger of quantitative research for the Americas at Lipper Inc., contends backtesting has its place but said the numbers can lose meaning when they reach back too far.
“You’re assuming that all the relationships in the marketplace that you use to presently build your models hold going backward and that may or may not be true. It’s less likely to be true the further you go back. If you start pushing things back two or three years it can be a little dicey,” Labovitz said.