An overlooked statistical concept shows why it’s so hard to beat a benchmark

However, I think you (and the author) are missing a much simpler explanation for why skew is important: for fixed expectation value, variance is intrinsically bad for risk adverse investors. If you want to try and pick winners and losers in the stock market, you will have to become less diversified to do so. Since there are many ways to interconvert between additional expected return and risk (e.g. insurance), inducing an exchange rate between the two, there is a quantifiable cost for active management even when random.

In yet other words: the cost of active management does not go to zero in the limit where the manager picks randomly and fees are ignored. Rather, as soon as you start picking you are accruing a cost through reduced diversification; you cannot come out ahead unless the size of your edge compensates for this non-zero cost.

So basically, the argument is that having an active fund manager is increasing your risk while also increasing your potential reward. The expected value is average, while the volatility is higher. While you might think this is an insignificant conclusion, I don't think this fact is obvious; most people aren't expecting that choosing an actively managed fund is increasing their risk.

(I agree there are plausible reasons for aggregate underperformance.)

Active investing will decline until active investing is profitable. The price of stocks is dominated by human emotions in the short term after all (ex: Enron cooked the books as early as 1996... but didn't go bankrupt till 2001). Holding onto a "short" position for 5+ years is basically unheard of, especially in the context of a Stock whose price is skyrocketing. (You probably would have been margin-called during that runup).

How many people would trust their "active investment manager" avoiding a hugely successful company with huge income growth? A quickly stock price and a good dividend? http://picker.uchicago.edu/Enron/EnronAnnualReport2000.pdf

The fact is: even with perfect knowledge about which companies are good and bad (Perfect 20/20 hindsight), its still incredibly difficult to make cohesive arguments for or against particular companies.

There might be rare occasions when that's justified (that company had major organizational issues when you worked there last month that aren't widely known, your technical knowledge means you know that hyped product probably doesn't work). But the rest of the time, you're just as likely to lose money overestimating how lousy the stock is as gain it.

The $100 chip might well be the one you (and many other stock pickers) think is trending downwards until it pulls off that unexpected achievement.

That said, you're right that excluding from a portfolio is a lower volatility strategy and much safer than shorting the "lousy" stocks. If I have reason to believe $TECHSTOCK is about to have their lunch eaten by new competitors and leave it out of my portfolio I slightly outperform the market if I'm right, and don't lose my shirt if Google buys it.

Part of the problem might be if some big gains come from apparent losers that somehow manage to pull a rabbit out of a hat.

Now a good manager should be able to look at the signs: most companies that will lose money have plenty of warnings in their various (10k, 8q or whatever) reports, not to mention you can look at an industry and see trends that hit the bottom line.

Should is key: many managers fail at this. However becareful when rating failures: the best managers long term have many years of below average performance - not just below market, below their peers are ultimately are not good. The best returns are from seeing a company that is doing bad now, but is going to turn things around in a few years (if you are a large enough you can even force this: buy enough of a company that you can change management). However this often means seeing that a turn around will happen sometimes in the next 5 years and waiting our 3 years of bad returns (and looking really bad!) before it happens.

I cannot tell who the best managers are, except in hindsight though.

This would be a fine equilibrium for wise unsophisticated investors, but unsophisticated investors are not all wise. Various human biases and market failures mean that there are an excess number of active investors (i.e., a number above the equilibrium value in an efficient market), many/most of whom are leeches. This is the state we find ourselves in today. The reason the number of active managers is decreasing is because wisdom is diffusing, but we are not done.

It goes without saying that "special expertise" involves a lot more than reading financial newspapers and considering yourself a smart person. It's about as hard to be able to compete at that level as it is to compete in professional sports. The big difference is that the heavy dose of randomness involved in investing allows more people to convince themselves that they are able to compete with the big boys, when almost none of them are. (The evidence that you can't compete with LeBron James at basketball is harder to ignore.)

Of course, plenty of professional fund managers claim to have that insight, but when they're charging higher fees for insights that on average don't beat the market, you should probably take the same view that you need enough knowledge to evaluate their trading strategy (or a strong reason to prefer a fund which claims to be aiming for low/high volatility) to consider them over the index...

Since the index has lower fees and the "winners" are disproportionately likely to have information advantages, beating an index is a competition in which the average joe can usually expect to fall behind.