This is my current understanding:
Mostly down to poor support for multiple cores in software, a lot of which have some kind of legacy heritage, and some tasks not translating well to running in parallel. Take video editing, for example - generally a single task the requires straight-line speed on one core, while the rest sit idle. Games are the most intensive for multi-core CPUs, but gamers usually buy top-end components, up to and including Xeons.
If anything, we tend to have a glut of CPU performance for day to day tasks. Even low-grade CPUs like Celerons are capable daily-use machines; some tech people (a friend included) see no reason to go beyond an i3. The multi-core paradigm shift around 2005-6 also massively increased CPU efficiency over anything that came before, reducing the need for existing applications to be reworked to exploit the extra cores. Intensive Javascript on sites can really drag your browser down, but even Chrome only runs one process per tab, and few people work with more than one tab at once. Multi-cores therefore tend to take care of background tasks, with only one core at a time doing all the heavy lifting. For one of the best examples of multi-core not being the ideal solution, one need only look at the Playstation 3. The 7-core (I think?) CPU is extremely powerful, but very, very difficult to program for, creating real headaches for developers trying to exploit its potential.
Multi-core might have drastically improved power efficiency, but it hasn't completely solved the heat generation problem. With the Prescott P4, Intel realised the core was generating so much heat it presented a serious problem to cool it. Cramming 10 of those on a single die would probably cause a China Syndrome. Adding cores allowed manufacturers to clock each core lower and still crunch a similar amount of numbers while reducing the heat output, but this sacrifices straight-line speed for single tasks. Ergo, AMD and Intel have to compromise for their consumer-grade products - enough straight-line performance that it doesn't feel slow for the current task, and enough cores to keep the throughput high. Looking at the current Xeon E7 v4 series, for example, increasing the core count above 4 results in a decrease in clock speed, likely entirely down to keeping the heat under control.
Add into this, laptop sales overtook desktops sometime around 2010, which require lower power consumption above all else. You could put a 16-core CPU in a laptop, but to keep power consumption realistic each core would probably end up clocked slower than a Pentium II. Further to this, a second shift occurred a year or two later where tablet and phone sales outstripped laptops. Mobile devices based on ARM CPUs are very power-efficient for all but the most demanding tasks. I know several people who've ditched laptops altogether and use Android or iPads exclusively. Essentially, current consumers have no need for the higher number-crunching capability of a desktop, which in turn means the traditional market for Intel and AMD has shrunk noticeably. ARM CPUs are starting to come standard with 8 cores, and it's difficult to find a current device with less than 4, all helped by the ARM architecture's power efficiency. Because they run an OS designed specifically for low power, not for general purpose, the cores can be clocked lower.
A lot of factors, but as noted, Intel and AMD are still improving their CPUs, just not by cramming more cores into the same die.