Evolution of indexing

The industry has rapidly developed over the past two decades

Debbie Carlson


To paraphrase an old advertisement, “indices, you’ve come a long way, baby.”

No longer just decided by a committee of business journalists, indexes come in many flavours: rules-based, fundamentally weighted, smart beta, thematic, direct and others, all to measure and present data to satisfy almost any investor.

While actively-managed ETFs are elbowing their way into the ecosystem, passive indices still dominate for their low costs and sheer quantity. And as technology advances, it leads to more ways to create indices, including using artificial intelligence (AI) in methodologies.

Indexing and ETFs grow together

Index-based investment products started with a mutual fund now known as the Vanguard 500 Index fund, but as Todd Rosenbluth, director of ETF research at CFRA, explained, the true rise of index investing coincided with the ETF market’s growth.

For the first decade of ETFs’ existence, market-cap-weighted index ETFs were largely all that existed (with the Invesco QQQ Trust (QQQ) a notable exception due to its tiered cap-weighting methodology).

The first investments based on fundamentally weighted indices arrived in 2003, with the now-named Invesco S&P 500 Equal Weight ETF (RSP); its underlying benchmark is still one of the best-known of these types of indices.

Christian Magoon, founder and CEO of Amplify ETFs, said the second generation of indices was inspired by investment professionals who used fundamental data in their strategies, which did not often align with market-cap or price weighting.

The ‘dot-com’ crash in early 2000 exposed the flaws in market-cap weighting, Magoon said: “I think that resonated specifically because of the big tech bubble. When that popped, it subjected the S&P 500 – and particularly the Nasdaq 100 – to some significant drawdowns.”

After the bubble burst

Using fundamentals versus price was not a new idea, he noted, but investors were not interested, because it underperformed market-cap weighting. However, the bursting of the ‘dot-com’ bubble increased acceptance of looking at indexes differently, and it spurred innovation.

Not long after the bubble burst, index providers started using academic research, specifically the factor studies from Eugene Fama and Kenneth French, who highlighted market anomalies around value and size, Rosenbluth observed.

Using fundamental weighting, smart beta strategies, whether single or multifactor, also ushered in new ways to think about indexing.

“This brought us to a world where indices are essentially created for ETF purposes. It is a significant shift from thinking about indices as ‘how the market is doing',” Magoon said.

While RSP is considered by many to be the first smart beta ETF, one of the first and most strongly promoted funds to rely on a fundamentals-focused smart beta strategy was the $4.4bn Invesco FTSE RAFI U.S. 1000 ETF (PRF), which relied on an index using a methodology developed by Rob Arnott and his firm Research Affiliates. PRF’s underlying index represented a turning point in smart beta indices, opening the floodgates for strategies like First Trust’s AlphaDEX methodology and, eventually, factor-based ETFs.

Thematics take the stage

Smaller entrants developed many of the newer ideas in indexing since they were not glued to the traditional index providers’ classification system, such as the Global Industry Classification Standard (GICS), Magoon explained. These newer indices took advantage of market opportunities. Indexers drilled down from sectors and styles to industries and knitted together companies based on a theme.

It is why thematics are interesting in the indexing world, Elisabeth Kashner, director of ETF research at FactSet, said.

“They take old indexing rules and break them. A thematics index looks for similarities that do not fit well in a traditional hierarchy. It is a different organisational scheme.”

That has allowed investors to bypass a broad-based technology ETF to find one based on a niche, such as 5G or blockchain. Magoon said Amplify’s digital payments ETF, the Amplify CrowdBureau Online Lending and Digital Banking ETF (LEND), contains some banking services stocks, software and IT stocks as well as investment banking stocks.

Thematics offer a connection between investors who want the focused exposure of individual securities with the diversification benefits of an indexed-based approach, he added.

Kashner noted that index providers now might look at different permutations to create an index, such as laying a factor on top of a classification scheme or overlaying an equal weighting.

“The easy ground has long since been covered. The new opportunities are in ever-smaller niches or ever-cleverer ways to combine things,” she added.

Not just for equities

Indexing is no longer just the purview of equities; fixed income is also getting the indexing treatment. Buying individual bonds means limited liquidity, diversification and transparency, but fixed income indexes solve that problem by offering diversification of securities, credit-quality, issuers, maturity and other aspects, Rosenbluth pointed out, and investors can trade them on an exchange.

While some investors may still be hesitant to use fixed income indexed ETFs because of unfounded fears that there might be too much money chasing these vehicles, he thinks that, as these get more customised, they may attract more users.

ESG and direct indexing

Direct indexing has been around for a while, and it has recently become popular as interest in environmental, social and governance (ESG) investing grows, and for its greater tax efficiency. Direct indexing lets investors buy individual equities, rather than using an ETF.

Jason Escamilla, CEO of Impact Advisor, said that for investors in high-tax states with taxable-account holdings, direct investing allows much easier tax-loss harvesting because he can trim individual stock holdings, rather than a basket of ETFs.

“You open up a wide range of tax opportunities,” he said, including moving appreciated holdings into donor-advised funds.

Additionally, the customisation is fine-tuned, which is particularly important for ESG investors who might want to give greater weight to certain ESG factors over others.

“The beauty of direct indexing is you can pick what your definition of responsible investing is, and build that into the algorithm,” Escamilla added, noting he can also offer this service to his submillion accounts.

Artificial intelligence: A mixed bag

In the past few years, some ETF issuers rolled out funds that rely on indexes built with machine learning, natural language processing and artificial intelligence, but to limited success.

Goldman Sachs teamed up with now-closed investment strategist Motif to launch five thematic ETFs that relied on machine learning. They were folded into a single fund after Motif’s shutdown. The Goldman Sachs Innovate Equity ETF (GINN) now has almost $400m.

More recently, ProShares rolled out the ProShares MSCI Transformational Changes ETF (ANEW) in late 2020; it currently has roughly $35m assets. The fund, like many newer thematic ETFs, uses natural language processing to seek out companies fitting within the parameters of its themes.

Merlyn.AI, however, has more successfully incorporated artificial intelligence into the indices underlying all the products in its suite of four ETFs. Some of them have gathered significant assets. The Merlyn.AI Bull-Rider Bear-Fighter ETF (WIZ) launched in 2019 and had $106m AUM in early 2021, while the Merlyn.AI SectorSurfer Momentum ETF (DUDE), which launched in the last days of 2020, had $123m.

Meanwhile, in Europe, the $344m Lyxor MSCI Disruptive Technology ESG Filtered UCITS ETF (UNIC) uses natural language processing and data analysis techniques to help identify companies based on the proportion of revenue that can be linked to the disruptive tech theme.

Mitigating the noise

The ETFs use signal processing, which includes the use of a double exponential moving average filter, to eliminate the noise around the momentum factor. Additionally, the firm uses genetic algorithms – algorithms inspired by Charles Darwin’s theory of natural selection – to evaluate hundreds of ETFs to select a portfolio of between three and eight ETFs. Machine learning helps the algorithm learn to improve ETF selection as market conditions change.

Scott Juds, chairman and CEO of Merlyn.AI, credits the early ETF success to having started the signal-generation strategy based on a subscription model back in 2010. User feedback allowed him to tweak the strategy well enough to eventually create an index. Running the models for a few years “warmed up the audience,” he said.

Julia Bonafede, co-founder of Rosetta Analytics, which uses AI-driven investment strategies, said employing AI and machine learning to build indices is still in its infancy, and may take some getting used to by investors. After all, she pointed out, multifactor risk models took some time to catch on before acceptance.

To start using AI-built indices, “investors are going to have to take the leap from a risk standpoint, and that depends on the governance structure of the fund,” she noted.

Economics of indexing

Even as indices evolve, some advisers in the US think the newer ones are not necessarily more advanced, but more specialised. Don Bennyhoff, investment committee chairman and director of investor education at Portfolio Solutions, wonders if the industry needs so many specialised indexed ETFs.

“It’s pretty hard to improve upon that plain vanilla type of approach,” he said. “Last I checked, the vanilla is still the favourite flavour of ice cream in the US.”

This story was originally published on