Thanks for your informative post. Related background on why that has happened by Dr. David Goodstein of Caltech from 1994:
https://www.its.caltech.edu/~dg/crunch_art.html "The period 1950-1970 was a true golden age for American science. Young Ph.D's could choose among excellent jobs, and anyone with a decent scientific idea could be sure of getting funds to pursue it. The impressive successes of scientific projects during the Second World War had paved the way for the federal government to assume responsibility for the support of basic research. Moreover, much of the rest of the world was still crippled by the after-effects of the war. At the same time, the G.I. Bill of Rights sent a whole generation back to college transforming the United States from a nation of elite higher education to a nation of mass higher education. Before the war, about 8% of Americans went to college, a figure comparable to that in France or England. By now more than half of all Americans receive some sort of post-secondary education. The American academic enterprise grew explosively, especially in science and technology. The expanding academic world in 1950-1970 created posts for the exploding number of new science Ph.D.s, whose research led to the founding of journals, to the acquisition of prizes and awards, and to increases in every other measure of the size and quality of science. At the same time, great American corporations such as AT&T, IBM and others decided they needed to create or expand their central research laboratories to solve technological problems, and also to pursue basic research that would provide ideas for future developments. And the federal government itself established a network of excellent national laboratories that also became the source of jobs and opportunities for aspiring scientists. Even so, that explosive growth was merely a seamless continuation of a hundred years of exponential growth of American science. It seemed to one and all (with the notable exception of Derek da Solla Price) that these happy conditions would go on forever.
By now, in the 1990's, the situation has changed dramatically. With the Cold War over, National Security is rapidly losing its appeal as a means of generating support for scientific research. There are those who argue that research is essential for our economic future, but the managers of the economy know better. The great corporations have decided that central research laboratories were not such a good idea after all. Many of the national laboratories have lost their missions and have not found new ones. The economy has gradually transformed from manufacturing to service, and service industries like banking and insurance don't support much scientific research. To make matters worse, the country is almost 5 trillion dollars in debt, and scientific research is among the few items of discretionary spending left in the national budget. There is much wringing of hands about impending shortages of trained scientific talent to ensure the Nation's future competitiveness, especially since by now other countries have been restored to economic and scientific vigor, but in fact, jobs are scarce for recent graduates. Finally, it should be clear by now that with more than half the kids in America already going to college, academic expansion is finished forever.
Actually, during the period since 1970, the expansion of American science has not stopped altogether. Federal funding of scientific research, in inflation-corrected dollars, doubled during that period, and by no coincidence at all, the number of academic researchers has also doubled. Such a controlled rate of growth (controlled only by the available funding, to be sure) is not, however, consistent with the lifestyle that academic researchers have evolved. The average American professor in a research university turns out about 15 Ph.D students in the course of a career. In a stable, steady-state world of science, only one of those 15 can go on to become another professor in a research university. In a steady-state world, it is mathematically obvious that the professor's only reproductive role is to produce one professor for the next generation. But the American Ph.D is basically training to become a research professor. It didn't take long for American students to catch on to what was happening. The number of the best American students who decided to go to graduate school started to decline around 1970, and it has been declining ever since. ..."
Also on that theme from a different direction by Philip Greenspun originally posted in 2006:
https://philip.greenspun.com/careers/women-in-science "The picture so far is pretty bleak. The American academic scientist earns less than an airplane mechanic or child support profiteer, has less job security than a drummer in a boy band, and works longer hours than a Bolivian silver miner. ... Does this make sense as a career for anyone? Absolutely! Just get out your atlas. Imagine that you are a smart, but impoverished, young person in China. Your high IQ and hard work got you into one of the best undergraduate programs in China. The $1800 per month graduate stipend at University of Nebraska or University of Wisconsin will afford you a much higher standard of living than any job you could hope for in China. The desperate need for graduate student labor and lack of Americans who are interested in PhD programs in science and engineering means that you'll have no trouble getting a visa. When you finish your degree, a small amount of paperwork will suffice to ensure your continued place in the legal American work force. Science may be one of the lowest paid fields for high IQ people in the U.S., but it pays a lot better than most jobs in China or India. ..."
That last part may be less true two decades later giving rising standard of living in China and elsewhere.
That said, like the lottery, the PhD system does work out for some people.
But in general, the PhD process is a deeply broken system that chews up most people who go through it. Freeman Dyson has written on this as well. One example:
https://www.science.org/content/blog-post/freeman-dyson-phd-...
""Oh, yes. I’m very proud of not having a Ph.D. I think the Ph.D. system is an abomination. It was invented as a system for educating German professors in the 19th century, and it works well under those conditions. It’s good for a very small number of people who are going to spend their lives being professors. But it has become now a kind of union card that you have to have in order to have a job, whether it’s being a professor or other things, and it’s quite inappropriate for that. It forces people to waste years and years of their lives sort of pretending to do research for which they’re not at all well-suited. In the end, they have this piece of paper which says they’re qualified, but it really doesn’t mean anything. The Ph.D. takes far too long and discourages women from becoming scientists, which I consider a great tragedy. So I have opposed it all my life without any success at all.""
Also related From the Village Voice from 2004 about Humanities PhDs who generally have it even worse than STEM PhDs:
"Wanted: Really Smart Suckers: Grad school provides exciting new road to poverty"
https://web.archive.org/web/20050228044956/http://www.villag... "Here's an exciting career opportunity you won't see in the classified ads. For the first six to 10 years, it pays less than $20,000 and demands superhuman levels of commitment in a Dickensian environment. Forget about marriage, a mortgage, or even Thanksgiving dinners, as the focus of your entire life narrows to the production, to exacting specifications, of a 300-page document less than a dozen people will read. Then it's time for advancement: Apply to 50 far-flung, undesirable locations, with a 30 to 40 percent chance of being offered any position at all. You may end up living 100 miles from your spouse and commuting to three different work locations a week. You may end up $50,000 in debt, with no health insurance, feeding your kids with food stamps. If you are the luckiest out of every five entrants, you may win the profession's ultimate prize: A comfortable middle-class job, for the rest of your life, with summers off.
Welcome to the world of the humanities Ph.D. student, 2004, where promises mean little and revolt is in the air. In the past week, Columbia's graduate teaching assistants went on strike and temporary, or adjunct, faculty at New York University narrowly avoided one. Columbia's Graduate Student Employees United seeks recognition, over the administration's appeals, of a two-year-old vote that would make it the second officially recognized union at a private university. NYU's adjuncts, who won their union in 2002, reached an eleventh-hour agreement for health care and office space, among other amenities.
Grad students have always resigned themselves to relative poverty in anticipation of a cushy, tenured payoff. But in the past decade, the rules of the game have changed. Budget pressures have spurred universities' increasing dependence on so-called "casual labor," which damages both the working conditions of graduate students and their job prospects. Over half of the classroom time at major universities is now logged by non-tenure-track teachers, both graduate teaching assistants—known as TAs—and adjuncts. At community colleges, part-timers make up 60 percent of the faculties. ..."
If all that is not bad enough, here is an even worse aspect of it all: "Disciplined Minds by Jeff Schmidt":
https://disciplinedminds.tripod.com/ "Who are you going to be? That is the question.
In this riveting book about the world of professional work, Jeff Schmidt demonstrates that the workplace is a battleground for the very identity of the individual, as is graduate school, where professionals are trained. He shows that professional work is inherently political, and that professionals are hired to subordinate their own vision and maintain strict "ideological discipline."
The hidden root of much career dissatisfaction, argues Schmidt, is the professional's lack of control over the political component of his or her creative work. Many professionals set out to make a contribution to society and add meaning to their lives. Yet our system of professional education and employment abusively inculcates an acceptance of politically subordinate roles in which professionals typically do not make a significant difference, undermining the creative potential of individuals, organizations and even democracy.
Schmidt details the battle one must fight to be an independent thinker and to pursue one's own social vision in today's corporate society. He shows how an honest reassessment of what it really means to be a professional employee can be remarkably liberating. After reading this brutally frank book, no one who works for a living will ever think the same way about his or her job."
One reason I am for a Universal Basic Income (UBI) is so everyone who wants it has a chance to live like an impoverished graduate student and pursue their intellectual dreams in cooperation with others of their choosing and without being exploited by an academic pyramid scheme.
Anyway, having passed through three PhD programs (in Industrial Engineering, Civil Engineering, and Ecology and Evolution plus essentially visiting another in Computing/Robotics for about a year), there is a lot more I could say on this (and have elsewhere).