First, the numbers thrown around are that once you own 15 diversified stocks you have reduced your portfolio risk by 70 percent. That's not bad. At this point your volatility may not be that different than a market index. Of course it's different for every set of stocks, but I think this is a safe-ish guideline. So in that sense you are not missing out a whole lot.
Second, and more importantly, it is very simple to put together 15 stocks with a low volatility, but also with very low returns. This is because the vast majority of the returns of an index come from a very small number of outperforming stocks. It's like the 90-10 rule. 90 percent of an index's performance comes from 10% of the stocks. Those aren't the exact numbers, but when you only pick 15 stocks, it is very likely that you will totally miss out on all outperforming stocks.
Here's an article that talks about this in more depth and uses real numbers: http://www.efficientfrontier.com/ef/900/15st.htm