Private industry needs to be incentivised to do more long-term R&D.
Corporate America just doesn't have the stomach to fund fundamental research with a payback varying from years in the future (optimistically) to never (pessimistically). In fact, we have exactly the opposite trend -- university researchers have taken on all kinds of applied research projects that would have been considered far too commercial in the past. In turn, the universities are looking to license out their IP portfolios and take stakes in companies founded by their students and faculty to raise more cash.
At Bell Labs the absolute best scientists could come together and just work, without having to deal with writing grant applications, teaching courses, or mentoring grad students (the three activities which now consume the vast majority of an academic researcher's time).
I am convinced that if Bell Labs was as strong today as it was 50 years ago, we'd have kilo-core microprocessors (and know how to program them), all our genomes would be sequenced already, and we'd all be driving electric cars recharged by ultra-efficient solar power stations. Or rather they'd be driving themselves. Maybe Google can help get us back on track.
The book, My Life as a Quant: Reflections on Physics and Finance, talks about why he left Bell Labs to be a quant on wall street.
I remember after graduating college. I was unsure what to do next. My profs wanted me to do PhD. My fiancee, a PhD candidate, was against it. The company I worked at was willing to pay for my masters in electrical engineering but again my fiancee made the argument since I was repulsed by large organizations that I would resent it. I would most likely end up working for those companies. He was right. This is what happened to my friends.
5 years ago, I was reconsidering a PhD, I thought comp sci or stats. So I dipped my toe and got a masters in stats - it was fun. Do a PhD? No. What for. Economically it would not make sense for me.
I would only do it if I found a fun topic such as modeling detection of prostate cancer (my masters project) but so far the idea of working for the immoral pharma industry including the unis and rest of them makes me ill.
I wish I learnt Marie Curie's secret on how to focus on the work and forget the rest. But I haven't.
My husband works as a computer science researcher (AI) - I KNOW I never want to do this. Pittance for ground breaking work done at excruciating slow pace with so many failures. The shit he puts up so he can work on the stuff that interests him - boggles the mind. I almost want to scream - get a bloody commercial job (effectively 2x salary) and use your own money to run your own tests. But he has seen what the prospects and the environment through the eyes of friends smarter than him and knows the true cost of politics. Smart man.
I would never advise anyone to do a PhD unless you are in love with the topic.
In one corner, we have engineers and scientists whose full time job is to solve problems and create new ideas; in the other corner, we have those whose full time job is to acquire money in any way possible - executives, Wall Street types, etc. It is easy to guess who will emerge financially victorious. Sadly, I don't think it can be any other way.
Most corporations are completely beholden to Wall Street these days, and stock traders can't evaluate long-term research initiatives.
Maybe the government and universities have squeezed the corporate sector out of research?
What would be interesting is to see some heavy quantitative studies done on the ROI of research in a corporate setting. If we could prove that R&D provides a decent ROI, I'd be willing to bet that the business types would be more willing to fund it.
I bet R&D departments benefit from economies of scale -- 5 PhDs can likely produce much more than one PhD at 5 companies, and a solid team attracts top talent -- which means the aforementioned big shots are the likeliest to produce meaningful research. Still, I would give a lot to work at a something like a "research startup."