(Bloomberg View) -- Professional equity managers -- of mutual funds, pension funds or hedge funds -- use a number of methods to try to beat their benchmark index. These broadly fall into one of two camps, “active" and “passive," and a debate over which gets better results has raged since at least the late 1960s.

Passive managers have been able to claim the upper hand by pointing to the well-known tendency of active equity managers to underperform their benchmark index.

Although this conclusion is widely accepted, there have been few attempts to find out why the index crowd outperforms. After all, it is one thing for active equity managers to fail to beat the benchmark index -- that could imply only a lack of skill to do better than random selection. But it is much harder to explain why most active equity managers fail to keep up with the benchmark index, a shortcoming that implies these investment professionals are doing something that systematically leads to underperformance.

The answer, we believe, lies in the fact that the best performing stocks in a broad index perform much better than the other stocks in the index, most of which perform relatively badly. That means average index returns depend heavily on the relatively small set of winners.

We developed a simple model that incorporates this idea. By randomly selecting a small subset of securities from the index, we maximized the chance of outperforming the index -- which validates the allure of active equity management-- but it also maximized the chance of underperforming the index, with the chance of underperformance being larger than the chance of overperformance.

Here is a simple illustration of our main idea:

Consider an index of five securities. Four (though we don’t know which) will return 10% and one will return 50%.

Suppose active managers choose portfolios of one or two securities and each investment is weighted equally. There are 15 possible one or two security "portfolios." Of these, 10 will earn returns of 10%, because they will include only the 10% securities. Just five of the 15 portfolios will include the 50% winner, earning 30% if part of a two-security portfolio and 50% if it is the sole asset in a one-security portfolio. The mean average return for all possible actively managed portfolios will be 18%; the median actively managed portfolio will earn 10%. The equally weighted index of all five securities will earn 18%.

In other words, the average active-management return will be the same as the index, but two-thirds of the actively managed portfolios will underperform the index because they will omit the 50% winner.

This figure illustrates our basic finding that random stock selection from an index is more likely to underperform than overperform.

That is, if the typical active manager’s picks are no better than a random selection from the benchmark -- a view many active managers might endorse, at least when judging their peers -- then we can expect those active managers to underperform the index.

Researchers have focused on the costs of active management as primarily the fees paid to the managers. Our results suggest a much much higher cost: the inherently high probability of underperformance that comes with attempts at stock selection, which disproportionately increases the chance of underperformance relative to the chance of overperformance.

Those allocating assets (such as pension funds and endowments) should revisit the assumption that the only cost of active investing compared with indexing is the active managers' fees. The benefits of identifying the best active managers may be greater than previously thought.

Subscribe Now

Access to premium content including in-depth coverage of mutual funds, hedge funds, 401(K)s, 529 plans, and more.

3-Week Free Trial

Insight and analysis into the management, marketing, operations and technology of the asset management industry.