What sort of solution could one imagine? Outside of post-doctorate positions, there's neither funding nor desire for the kind of research that you need to further an academic career.
1. http://www.tau.ac.il/~algazi/mat/Goffman--Cooling.htm
2. http://kieranhealy.org/blog/archives/2002/10/31/cooling-the-...
"My own knowledge comes from universities, both in the United States and Britain. In both countries, the last thirty years have seen a veritable explosion of the proportion of working hours spent on administrative tasks at the expense of pretty much everything else. In my own university, for instance, we have more administrators than faculty members, and the faculty members, too, are expected to spend at least as much time on administration as on teaching and research combined. The same is true, more or less, at universities worldwide." http://www.thebaffler.com/salvos/of-flying-cars-and-the-decl...
Articles like this indundate the reader with chart upon chart. High-res pictures with a scope too narrow to find serious solutions. What do these charts leave out?
At least in Computer Science, the alternative is to simply go into industry at a company that will let you continue publishing, and get a paper out every year or two. Lots of people have successfully made the transition back into academia after a post-PhD stint in industry.
You won't be able to net a position at a top R1, but plenty of universities (and esp. colleges) would be happy for someone a few years out from a solid Ph.D. but with a strong industrial background and track record of (perhaps ad junct) teaching and/or moonlight researching. Especially if you stay in industry long enough that you don't need a six figure salary.
Congratulations on successfully navigating that transition, BTW.
But why not? If academic work is for the benefit of mankind, there should be a desire in society for that kind of research[0].
This desire is only muted because of commercial interests having increased influence (directly and indirectly) over academia.
[0]Perhaps there is a mismatch in what we mean by 'kind of research' here. But this is a political point as well - the kind of research that has long-term benefits to society should be exactly the kind of research that furthers academic careers.
I was preempting a response that I often hear when this debate comes up, which is that "the market" will fix the problem of underemployed PhDs. This often turns into an argument that research that has "practical value" (as judged by the general public) is somehow more important than that which does not. On Hacker News I sense that it's common to read "PhD" as "STEM PhD," and the response is "it's fine, they can just find positions in the industry."
This logic tends to fall short when applied to the humanities in general, and external (non-university) funding for humanities research is shockingly less than those in STEM fields. The National Endowment for the Humanities has an annual budget of $167 million, and the National Science Foundation, compared to $7 billion for the NSF (numbers from Wikipedia). Humanities research often tends to get mocked in debates on public funding: "Why are we, the taxpayers, paying this person to go look through archives in Germany looking for fragments of music by Robert Schumann?"
The humanities do not, in general, have "practical application," at least if practical application means monetary value (as it often does). As 1stop notes in a sibling, humanities research is often research for the sake of culture, or for the sake of research itself. I got into my field because I love it, but also because I think there is inherent value in knowledge. The number of people who will read my dissertation could likely all enjoy a meal together, but for those few people it matters deeply.
So there is some long-term benefit to society for humanities research (other people, likely also scholars, can benefit from it later), but it's not a benefit that's easily quantified. In today's society, this is a problem without a clear solution, and it's an attitude I see a lot on HN (with constant talk of valuation and whatnot).
I feel comfortable taking the opposite position - if work with a practical application is worthwhile it should be commercially viable and we don't need to make any special effort to fund it academically - corporations and capitalism will surely cover it.
On the other hand, the humanities require investment that corporations are less willing to make - therefore the public should make more effort to fund these.