With index investing all the rage these days, it's easy to forget that today's best-known stock market indices were not designed with the idea that people would invest in them directly uppermost in mind.

Most were created by journalists or publishers as a way of providing summary information on a country's biggest stocks - a useful snapshot of the health of the corporate sector, and a barometer of animal spirits among investors. Most such indices are weighted by market capitalisation (market cap) - in other words, the bigger the stock the more clout it has in the index.

That makes sense if you're looking to build a 'Top of the Pops'-style index of the country's biggest stocks. But it doesn't intuitively feel like the best approach for an investor. One oft-repeated and not unreasonable criticism of passive investing is that by tracking a market-cap weighted index - via a 'plain vanilla' exchange-traded fund tracking the S&P 500 or FTSE All-Share, for example - you are effectively buying high and selling low. As a stock becomes more popular and more expensive, its weighting rises in the index, and so you buy more of it, while off-loading the less popular stocks as they grow cheaper.

In practice of course, history shows that this works pretty well, and that as long as you can do it cheaply enough, 'merely' achieving the return on the index is sufficient to beat the majority of active fund managers. Hence the massive and growing popularity of passive investing. But it does feel as though there should be a better way - a way to get returns that consistently beat the market, just by tweaking your approach. That's where smart beta comes in.

The idea behind smart beta is to build a better index. As we all know, and the regulator always emphasises, past performance is no guide to future performance. But it's also the only one we've got. And history shows that there are certain characteristics - what the industry calls "factors" - that help stocks to outperform, given a long enough period of time. Smart beta providers create indices that use these factors to screen for stocks that have these characteristics. The idea is that if these factors continue to work, then you should - over the long run - outperform a bog standard market-cap weighted index.

Holy grail or a poisoned chalice?

Clearly, there's a lot of money to be made here. Smart beta is the holy grail - a mechanised strategy that actually delivers the returns that active managers promise to deliver, but rarely achieve. It's an appealing proposition to investors, certainly. The amount of money in smart beta funds has trebled since 2012, according to Morningstar, and it looks set to hit $1tn worldwide by 2017. Better yet, you can charge more for it than for a standard ETF, while keeping both charges and costs lower than for an active fund.

As a result, there's been something of a gold rush on factors. Academics are falling over themselves hunting for new anomalies in the stock market, that can be used to market new smart beta approaches. (They then spend even more time arguing among themselves over whether their findings prove that the market is efficient or inefficient, but that's a discussion for another day). By some estimates, there are now 500-odd factors to be exploited.

There's just one problem: most of them don't work. One team of researchers, led by Antti Suhonen, a finance professor at Aalto University in Finland, looked at 215 strategies across five separate asset classes. They found that the theoretical outperformance of a typical 'new' factor collapsed by nearly three-quarters when it was applied in the real world. In other words, it looked great in the backtests leading up to publication, but as soon as it was applied to a live market, the "secret sauce" apparently evaporated. The deterioration was also more pronounced for complex strategies compared to simple ones. This echoes previous work by smart beta pioneer, Research Affiliates, which found that once a factor has been identified, its average outperformance roughly halves.

This is not - as you might believe - down to investors eyeing the academic papers and then arbitraging the strategy to death (which is what markets are meant to do, of course). The real issue is 'data mining'. These days, if you are conducting backtests using powerful computers, it's very easy to tweak your strategy continually until you find a variation that would have delivered great returns. It looks significant on paper, but in fact all you've done is the equivalent of going through past lottery ticket winners and changing the numbers you put into your search, until you find a winning ticket - and then suggesting that those numbers will deliver huge returns in the future. Alternatively, some of the strategies may well work in theory, but only with an impractically small amount of money - for example, strategies that exploit inefficiencies among micro-caps.

None of this means that smart beta is fundamentally flawed - indeed, it's a very compelling idea. But as with all things related to the finance industry, the risk is that fund providers take a good idea and milk it to sell investors poorly-designed, highly-priced products that fail to meet their expectations. The answer (as always) is to understand what you're investing in and how it plans to make its money before you buy any smart beta ETF. There are three main factors that have been consistently found to 'work' historically. There's momentum - buying stuff that's going up, and selling stuff that's going down. There's value - buying stuff that's cheap. And there's small caps, which tend to beat larger caps. That's not to say that other factors won't work, or that these three will always work (value has struggled for over a decade now). But if an ETF is based on something more esoteric than these three, then take a good hard look at its holdings and how they match the rationale behind the strategy (for example, Apple seems to fit the bill for almost any smart beta ETF you can imagine). And if you can't wrap your head around it, stick with plain vanilla.