Why Beating the Market is so Hard

Introduction

Investing is aspirational. Nobody buys stocks in the hope of losing money. The primary goal of individual investors and fund managers alike is to achieve a seemingly elusive feat: beat the market. Unfortunately that's easier said than done. Let's explore the various factors that make this such a difficult task.

Let's Get the Maths Out of the Way

In stock markets, a significant portion of total returns is often attributed to a small number of high-performing stocks. For instance, in a market of 1,000 stocks, if just 50 of them account for the majority of gains, missing these in a portfolio can severely impact its performance relative to the market. The challenge lies in the improbability of consistently identifying these outperformers in advance. For example, if the total market return is 100, generated by 100 stocks, and 10 stocks contribute 70% of this return, not holding these 10 in your portfolio could substantially diminish your returns, as you're left competing for the remaining 30% return with the other 90 stocks. This distribution pattern makes it mathematically challenging for active strategies to outperform the market, often making a strong case for diversified or passive investment approaches. Obviously, the more you diversify, the stronger the arguement to simply invest in a low cost market tracking fund.

It's What You Know That Counts

Information asymmetry, where one party has more information than the other, has long been a pivotal element in financial markets. With the advent of the internet, information asymmetry has drastically diminished. Today, individual investors have almost instantaneous access to a wealth of information that was once the exclusive domain of professional traders and large institutions. Unfortunately, we're now faced with a different challenge - filtering out the information that matters, before it becomes common knowledge.

By the time information becomes mainstream, the price will have already adjusted and the opportunity lost. Investors who follow the latest news cycle or hot stock tips, probably miss the material opportunities to beat the market.

Impact of Transaction Costs and Fees

Transaction costs and management fees are often overlooked yet crucial factors in the pursuit of market-beating returns. Every trade comes with a cost – whether it's brokerage fees, bid-ask spreads, or taxes – and these costs can accumulate significantly over time, especially for active traders. Furthermore, for those investing through mutual funds or managed portfolios, management fees can take a substantial bite out of potential gains.

These expenses effectively create a performance drag on investment returns. Even if an investor or fund manager makes trades that outperform the market on a gross basis, once these costs are factored in, the net returns may no longer surpass the market average. This reality makes the challenge of beating the market even more daunting, as it requires not just equaling market performance but significantly outperforming it to cover these additional costs.

Your Own Worst Enemy

The realm of behavioral finance offers insightful perspectives on why beating the market is an uphill battle. Key psychological factors, such as herd mentality and overconfidence, play significant roles in investment decisions.

Herd mentality refers to the tendency of investors to follow and mimic what others are doing. This behavior often leads to buying stocks at their peaks (when everyone is buying) and selling at their lows (when panic sets in), which is contrary to profitable investing strategies. Peter Lynch, who managed the Magellan fund and delivered an average performance of XXX per annum noted that the average investor in the fund only returned 7% per annum. Exactly because their biases led them to invest after great performance and withdraw after poor performance.

Overconfidence is another critical stumbling block. Many investors overestimate their knowledge and ability to predict market movements, leading to riskier bets and more frequent trading – behaviors typically associated with lower returns. These biases result in decision-making that deviates from rational, disciplined investing, hampering an investor's ability to consistently outperform the market.

Market Volatility and Timing

Market volatility and the challenge of timing the market correctly are formidable obstacles. Financial markets are volatile, with prices fluctuating due to various factors like economic news, geopolitical events, and investor sentiment. This volatility makes it extremely difficult to predict short-term market movements accurately.

Furthermore, the timing of both entry and exit from the market significantly influences investment returns. Even if an investor correctly predicts a market trend, entering or exiting the market too early or too late can lead to missed opportunities or losses. The risks associated with market timing add to the difficulty of consistently achieving returns that outperform the market.

Professionals Eat Amateurs Lunch

Professional investors and institutions have significant advantages, including experience, access to sophisticated research, advanced trading technologies, and highly skilled investment teams. Professionals can analyze and interpret market data more efficiently, and they often have the resources to utilize complex trading strategies that are beyond the reach of average individual investors. This means that for every poorly thought out trade idea, or behind every news article lies an market participant ready to capitalize on the mistake.

This competitive landscape creates an asymmetric battlefield where individual investors without a tested strategy may find themselves at a disadvantage.

In it for the Long Haul?

The difficulty of beating the market can vary greatly between short-term and long-term investment strategies.

Impatience tends to shorten the average investors time horizon. A short term mindset leads investors to over-trade, or take on undue risk - both of which are correlated with poor investment returns.

In contrast, long-term investing involves holding securities for extended periods, often years or decades. This strategy benefits from compounding returns, lower transaction costs, and the ability to ride out market volatility. While it's still challenging to beat the market, a disciplined, long-term investment approach tends to yield more favorable outcomes compared to the high-risk and often erratic nature of short-term trading.

The Stats Don't Lie

If you need any more convincing that beating the market is difficult, just look at the performance of actively managed funds over various time horizons.

The longer the horizon, the harder it is to consistently beat the market. When looking at the results over a 20 year period, and astounding 96% of active managers underperform the market.

Hard, Not Impossible!

Based on the information above, one might easily conclude the beating the market is an impossible task. It's true that beating the market is difficult. Difficult, but not impossible. There are numerous examples of investors who've successfully and continuously beaten the market. Some of these are accounted for in the excellent market wizards series by Jack Schwager. For a detailed account of what it takes, I highly recommend checking this series out.

For most investors, there is little reason not to adopt a passive investment approach and invest in a market tracking index fund. Unless you have what it takes to beat the market, this strategy will likely perform better than a selection of arbitrarily selected stocks. Alternatively, you can try for a hybrid approach and adopt the barbell investment strategy.

Conclusion

The quest to consistently beat the market is a challenge that the average investor finds themselves underprepared for. Understanding the various aspects that make beating the market difficult is critical for investors who want a chance at doing so.

While the aspiration to outperform the market is understandable, it's essential to have realistic expectations and adopt informed investment strategies. For the average investor, there is good reason to adopt a market tracking passive investment strategy.

For those interested in beating the market, recognizing the inherent difficulties and approaching investing with a well-thought-out, disciplined strategy are a prerequisite for successful investment outcomes. The key is not just in seeking to beat the market but in understanding and navigating the complexities of yourself as an investor.