Antimarket opportunities
blog.kenperlin.com
blog.kenperlin.com
Some companies -- Google, obviously, or Bell Labs back in the day, or companies run by well-heeled visionaries like Musk -- are willing to sink cash into high-risk research, but in general Google is going to see more returns out of incrementally improving AdWords rather than developing self-driving cars.
The other thing to think about is that R&D doesn't just "skate to where the puck will be," it actually places the puck on the ice. It takes fundamental and/or high-risk research to create these kinds of opportunities, which means that, in a world where markets solely dictate research objectives, a lot of potential advancements will be left on the table as long as companies have other, profitable, avenues of investment. Which is why we need government funding, though not government control, of research -- but that's probably a discussion for another time.
Without accountability (which is what you mean by 'government control'), all you will have is crony funding, and a downward-spiraling quality of work as effort goes toward scientific rent-seeking instead of real scientific product. This problem is inextricable. In order to have accountability, you have to have benchmarks, which really means that the only type of science funding the government is consistently good at is translational, with defined goals. A good example is the moonshot.
If your claim is that government will politicize science, well then, the problem is with demanding that government fund science in the first place. It is a ridiculous position of entitlement to demand money, obtained from taxpayers who have little choice to comply, with nothing in exchange.
For an excellent counterexample to your assertion, consider Peter Mitchell. He created an private, nonprofit science research organization very much dedicated to a 'basic science' question and solved a fundamental question in biochemistry: How does the cell generate energy? Although there is no way that industry would have funded this, there was also no way that government would have funded it either. He won a Nobel Prize for his work.
Do not make the mistake of conflating 'private' with 'for-profit'. This is a false dichotomy that overlooks important social efforts, such as the Polio Vaccines.
> If someone else thinks that what you are doing has value (even in the long range), there is a market for that to be done and you will get paid
No, because the monetary market doesn't do a good job of predicting scientific discovery. This makes sense: if someone was already confident that a particular future discovery would happen then it wouldn't be much of a discovery, would it?
That's not a problem with the market, that's a problem with people, and there's nothing you can do about that. The market matches supply with demand. And besides, 'ability to predict' is also not necessarily coupled to demand. There are people who are willing to pay to see science even if they can't predict success.
There are also a lot of people who like saying that 'something needs to be done about funding' but when when pressed: "well why don't you help pay for it." they don't reach into their own pockets. These people are part of the problem.
What's the motivation to work on a potentially big idea rather than something you personally find great interest in?
Wouldn't it be best to just work on whatever interests you most, or is there such a wealth of branches you could go off into, that one feels the need to narrow it down and this is one of the ways to do that?
The two aren't contradictory. All the academic folk I know work on ideas that they find interesting. And some of those things are big ideas.
The difference is that there isn't a short-term economic benefit attached to the vast majority of those ideas.
Let’s start with your career as a doctoral student. If you personally find great interest in your research, but nobody else does, then you will find it very hard to get published (except in bullshit fake conferences and fake journals that accept any author who pays the fee) and you will not be able to persuade your committee to accept your dissertation, so you will not get a doctorate and will not graduate. Also, if no PIs at your institution think your research is interesting, you will not get funded as an RA, and you will have to devote a lot of your time to teaching instead of research, or you will have to pay tuition. If your research is in a field that requires access to equipment (e.g. chemicals, supercomputers, nuclear reactors, oscilloscopes, pencils) then not getting funded as an RA also likely means you don’t get access to the university’s equipment, so you have to buy your own, which may completely stop your research or just slightly annoy you, depending. If your research involves experimentation on animals or humans, the IRB or AEC is not going to approve your application just because you personally find great interest in the research; you have to convince them that it’s worth the potential harm to the experimental subjects. Finally, and perhaps worst of all, if you personally find great interest in your research, but nobody else does, then only your ideas are going to go into it; nobody else will contribute, and so you will progress maybe 1% as fast as if you were surrounded by a community of like-minded peers.
But suppose you work on stuff your advisor likes until you graduate in seven years, and then another four years in postdocs, and then you publish a bunch of papers as an assistant professor (on subjects that referees find so interesting that they recommend publication) and lots of other people find your ideas so interesting that they cite your papers so much that you are eligible for tenure, and then you actually get tenure because you didn’t grab any students’ asses or fail to actually physically be present when you were suppposed to be teaching or anything like that. So now, after 7+4+6=17 years of working on ideas that other people thought were interesting, you have tenure and you can, if you so choose, work only on things that you personally find great interest in.
At least in theory, and modulo the concerns about IRBs and AECs above. But your department is going to be a lot happier with you if you are working on things that you can find funding for, and if your ideas are attracting brilliant applicants to the department’s doctoral program, so that they can then admit those applicants and coerce them into working on the ideas they (your colleagues) find great interest in, in exchange for a lottery ticket that might pay off with tenure. In practice, though, tenured faculty tend to spend a great deal of time writing grant applications to persuade grantmakers that they will work on ideas that the grantmakers find interesting, giving talks to persuade their colleagues to be interested in their work, and teaching classes with at least the ostensible objective of persuading students to be interested in their work. Also, sometimes you have to ask grantmakers and grantmaker-like agencies not only for money but also for permission; nuclear research in the US, for example, requires that your research reactor be overseen by the DoE.
So, those are some of the structural incentives in the US academic system to work on things that other people in your field think are potentially big ideas, instead of just things you personally find great interest in. You know all this bullshit about how the web is an “attention economy”? Academics have been desperately struggling for attention for centuries.
And, yes, you need to be interested in what you’re investigating, too; not least because, if you aren’t, you’re going to have a hell of a time getting anybody else to pay attention, and you almost certainly aren’t going to make it through grad school.
Your last question is about right as well. All the most successful academics I've met are incredibly creative people, with tons of ideas. If you have the choice of working on two equally interesting ideas, why not pick the one that's likely to turn out to be big? That will also help attract smart colleagues, which is, as much as anything else, one of the big perks of academia (not that you can't get that elsewhere for better salary and job security).
I can't think of much, offhand. It's even harder to skate where the proverbial puck will be twenty years in the future.
Inventing a piece of the puzzle is hardly a guarantee, but it has worked out well for some.
http://en.wikipedia.org/wiki/List_of_most_expensive_domain_n...
20 years ago they didn't, but the more interesting question is what's the thing that's available cheaply today that people will be buying in 20 years. If you have some ideas for this in the unethical space I am genuinely interested.
p.s. thanks for the vote of confidence though :)