Randomness, Efficiency, and Opportunities from Trends

Many studies on hedge fund performance have concluded that it is impossible to achieve superior returns in the market solely by applying rules or formulas, as these methods do not yield better results than a randomly diversified investment portfolio. This has prompted financial theorists to seek answers to a crucial question: Why does a well-functioning financial market eliminate excessively high returns over long periods? The Efficient Market Hypothesis (EMH) provides one such explanation.

According to this hypothesis, at any given moment, all new information is immediately absorbed by market participants and reflected in asset prices. This leads to the phenomenon of the random walk, where price movements are erratic and unpredictable. Studies on market-wide price indices, as well as individual assets, suggest that their price charts resemble a sequence of random coin tosses.

While I do not entirely agree that markets are perfectly efficient, I also do not reject the idea that prices move randomly. However, this randomness is not mystical; rather, it stems from our limited computational power and the lack of sufficiently robust models to capture it. In reality, everything around us is influenced by randomness, originating from the smallest quantum particles. According to modern physics, fundamental particles behave randomly, yet when they combine into larger systems, they follow discernible laws. A prime example is entropy, a scientific measure of disorder and uncertainty, which, despite being seemingly random, follows a strict rule: it can only increase over time.

I believe market prices share this nature—while appearing random, they are still governed by underlying principles. Price movements form a sequence of events structured in time, and these events can be represented by a probability density function. If we start from such a function, we can generate multiple different random sequences. Thus, in finance, constructing a model to capture these random events is essentially a process of identifying the probability density function behind them.

If we take the daily closing prices of a stock (or any asset) over a long period, arrange them based on their percentage change from the previous day (from the largest drop to the highest increase), and plot the frequency distribution, we obtain a return distribution. In this perspective, every price chart is essentially a random distribution of price changes over time.

Now, let's look at the illustration below. Proponents of the Efficient Market Hypothesis build their models under the assumption that financial market returns follow a normal distribution (the blue curve with a higher peak). This implies that returns are symmetrically distributed around the mean, meaning the market typically fluctuates close to its previous day's price with minimal extreme movements.

However, real-world data suggests otherwise. Market prices experience extreme events far more frequently than what a normal distribution predicts. Based on extensive statistical analysis, I have observed that actual return distributions exhibit fat tails (represented by the orange curve with a lower peak).

I have analyzed various financial assets, including stocks, commodities, and derivatives, using the same methodology to generate multiple probability density function plots with Python. During this process, I examined different asset classes and divided the data into smaller time intervals of varying lengths. In the one-minute timeframe, the dataset grew to tens of millions of data points, significantly increasing the reliability of the findings.

Although we can never achieve absolute certainty, having a vast amount of data allows us to estimate probabilities with higher precision. In trading, nothing is guaranteed, and fortunately, we do not need certainty to make money. After analyzing a wide range of assets, I have yet to find any asset that truly follows a normal distribution—almost all of them exhibit deviations in the tails of the distribution.

This does not necessarily disprove the Efficient Market Hypothesis, but it offers a different perspective on how the market operates. I believe that markets are efficient and move randomly within a certain framework, but there are periods when they become inefficient, and these inefficiencies present opportunities for profit.

Dividing Market States

Based on the data I have collected, I have developed an alternative perspective on market behavior and classified its randomness into two distinct distribution states:

This concept was first introduced by Nassim Nicholas Taleb in The Black Swan (2007). Taleb categorized randomness into two domains:

• Mediocristan, where randomness is "normal" and follows conventional statistical rules.
• Extremistan, where extreme events dominate and have disproportionately large impacts.

If we analyze price changes separately in these two regions, we obtain different types of distributions:

• Mediocristan corresponds to the normal distribution assumed by Efficient Market Theory. In this state, prices frequently revert to equilibrium, move randomly, and are generally unpredictable. Consequently, it is difficult to generate significant profits.
• Extremistan, however, exhibits fat-tailed distributions, where prices move in clear trends. This is where extreme events dominate and have a significant impact on overall market outcomes.

By isolating these two different market states, the fat tails in the Extremistan distribution become much clearer compared to the overall price distribution. This means that if we can filter out market conditions where prices exhibit trends, we can take advantage of these fat tails to generate profits.

The Most Powerful Approach to Making Money in Financial Markets

This implies that, in the random environment of financial markets, the most effective and powerful method to make money is through trend-following strategies. When the market enters an Extremistan state, where large price movements occur and randomness gives way to clear trends, traders can exploit this behavior for profit.

Rather than attempting to predict small, unpredictable fluctuations in Mediocristan, where price movements remain within a normal range and profits are difficult to sustain, focusing on Extremistan—where extreme movements occur more frequently than normal distribution models suggest—offers a significant edge.

Thus, trend-following strategies, which capitalize on identifying and riding these inefficiencies, are not only practical but also a fundamental approach to optimizing long-term profitability in financial markets.