Why Innovation Prizes Fail
worksinprogress.co
worksinprogress.co
You may want to incentivize discoveries by financing proven talent, but that approach ignores the fact that being successfull has an important component of luck.
There's a recent economic model [1] studying the effects of sheer luck in acquiring wealth; and it shows that the richest people end up being those who accumulate a streak of lucky breaks, independently of their talent (above a minimum).
The model predicts that the best strategy for getting talented people to deliver results is giving a small amount of funding to everyone equally. This way, you maximize the chances that someone will start a project with the right approach, AND that are least some of them will have the lucky breaks required for it to flourish.
[1] https://www.inc.com/chris-matyszczyk/so-youre-smart-but-your...
That's largely true but not very relevant because most innovation prizes are independent of past performance. Your criticism might be applicable to one of the solutions the author proposes (essentially grants to people pursuing ideas, which presumably would be based on past performance) but not to innovation prizes as such.
I don't have a solution except to run a brainwashing campaign on funding sources to convince them that luck is a thing (maybe put it as "luck needed = 1 - risk") and that multiplying many risky experiments increases the probability that one of them will get lucky.
Those are not incompatible. You can have your Noble Prices for recognizing and publicising outstanding achievements, without any expectation that they will create any further equivalent outcome.
And at the same time, have a general talent program financing anyone who applies, expecting most of it to be non-refundable losses, but with the expectation that a couple of them here and there will pay off and recoup the cost of the entire program. Isn't this what first-round VC is in essence?
How does that protect against people squandering the funding? They need some incentive to try to achieve something with it. That's what an innovation prize offers -- a reward for performance. "future performance" is the candidate's problem, not the funders.
This is how sports and stock markets work, relatively effectively.
Yes, but remember, Fortune favors the prepared.
I was very lucky many times in my life, but I didn’t follow that with work and grit. Elon Musk and Jeff Bezos did. So they are where they are, and I am where I am.
Elon Musk was born into an already wealthy family and failed his way upward.
Jeff Bezo's wife was already a millionaire when he started Amazon
Neither of them had "hard work" or "grit", despite the whitewashing both of them have done of their rags-to-riches story. They are where they are because of generational wealth and familial connections.
> Fortune favors the prepared.
A thing shown by the study is rather that fortune flees from the lazy one. Above a certain threshold of talent (defined in a way that includes effort and know-how), more talent does not imply better results.
This is a far superior summary of the aforementioned study. It also explains how the computer-simulated experiment was conducted
>All agents began the simulation with the same level of success (10 "units"). Every 6 months, individuals were exposed to a certain number of lucky events (in green) and a certain amount of unlucky events (in red). Whenever a person encountered an unlucky event, their success was reduced in half, and whenever a person encountered a lucky event, their success doubled proportional to their talent (to reflect the real-world interaction between talent and opportunity).
The issue with this study I always had was how contrived an unlucky event (halves success) was. Do these parameters actually map to the real-world? If anyone with actual chops in this field could comment on this, it would certainly be very helpful.
Modern innovation prizes, like Elon or say Twitter's attempt to figure out how to monetize comes with unsurprisingly draconian IP shackles. You work becomes there and you can't make money off of it. You get a relatively small reward for a huge boon to someone else. Companies spawn out of innovation prizes, not free ideas for the sponsors to profit off of.
Right now, a drug company (like Pfizer) will spend a billion dollars, then if successful, they have a patent and have to recuperate their investment before expiration (meanwhile, patients and their insurance providers are paying the premium, and those who can't, can't get the drug). But (depending on where you stand politically), these drugs have some public good, and it could be more direct to award prize money, then remove any restrictions on the drug after discovery.
https://www.herox.com/blog/428-history-of-challenges-the-ort...
I haven’t investigated the effectiveness of modern DARPA Challenges but they seems like a good value for our defense dollars, where we spend trillions.
Weird to phrase it as "need". I look forward to your proposal for replacing supply and demand determine wages with ones based on number of dependents.
In the DARPA had two tracks for their competition, with competitors in one of the tracks being granted $1 million to fund their entries. Now, you had to qualify to get that funding so a reputation and some prior work was needed, but the consolation prize was you were able to advance your career whether you won or lost. That is why it worked. You can't just dangle a giant carrot at this level and expect people to bite if it's a zero-sum game.
Let's say I invest millions and years of my time to get the $100 million prize, and someone gets it right before me. What's my consolation prize? If it's absolutely nothing, then I'm not going to risk my career on that bet, and probably the person who would actually crack that egg won't either. Moreover, they might still try but it will be through tested funding channels.
Isn't that exactly how startups work? A $100 million prize is reaching into baby unicorn territory (a fully grown unicorn is "only" $1B in value), so why wouldn't the prospect of winning the grand prize be enough to attract funding?
The DARPA grand challenges recognized that this incentive structure doesn't work well. What they did instead was they held 3 smaller prizes instead of one large one. This meant that development was incremental over 5 years, which they also funded.
What they did in those 5 years was take driverless cars from running off the road into rocks in the middle of the desert in 2003; to converging on an architecture that could reliably navigate an urban environment by 2007, an architecture which is now the basis for all commercial driverless car efforts. If they had offered just a single lump sum for a demonstration of such a car, I think the entire industry wouldn't be where it is today.
Then, within the institution, there will need to be internal competitions as to see which proposal(s) gets the funds to go after a prize.
There is nothing unique about me above. There are a lot of smart people around who are not doing research. Basic research is important to progress humanity, but for every person in basic research thousands are needed in the more applied areas to take known research results and apply them to life thus making our life better.
I agree. Personally, it's a big reason why I went to industry to do research.
> Maybe I misunderstand you but institutional funding does almost never enable research.
Mostly disagree. Two reasons: other institutional funding sources (e.g. teaching positions, personal or company grants, fundamental research grants, ...) and post-docs or profs hustling to shelter phd students from most of that overhead.
Despite the incredible inefficiencies inherent in competitive research grants, research labs can scrap together funding with some form of continuity.
What you do get often is an 'accordion effect', where a lab can swell in numbers on the coattails of few successful phd students and series of impactful papers, but at some point there's a crunch when it cannot lock in the same number of new research projects.
Your colleague is right that being a PI in a research lab is a lot like being a startup founder. But we shouldn't take that to mean that research labs should be startups, with goals like establishing a profitable revenue stream that will assure a self-sustaining operation. That's just not the purpose of research, and trying to make it as such would change the kind of research being done.
The other thing is that like a startup, research labs get a significant chunk of "startup capital" in order to get their research agenda off the ground. Depending on the agenda, this can range from 10s of thousands to a million dollars or more.
Third, there is other institutional funding that goes beyond that. It's true that most funded research is through grants, but at the same time it's hard to get grants without some prior research results. That's where institutional funding comes in -- to give you a little boost to bootstrap some results that you can then take to a funding agency.
> So "researchers" in research institutions spend most of their time looking for funding.
I think I know why you put "researchers" in scare quotes, and that is because of a common perception that "research" is limited to the technical work of running an experiment and writing papers (please correct me if I'm wrong). But I disagree with this view, and instead I consider research as also including getting the money.
Why? It has to, because getting the money involves setting clear research objectives, communicating those objectives to others in an understandable way, forging alliances with other researchers, navigating regulatory obstacles, hiring competent research personnel, designing research experiments, and most importantly managing the project over a decade or more without the whole thing imploding. That is research, because if you don't do those things you're just doing a "project". Getting something into the world and getting buy-in from external agencies is important precisely because there is no revenue stream and no customers who provide feedback to your venture.
Sitting down and doing the coding, running experiments, or even writing the paper is important to the scientific research process, but it's still just a part of it. Certainly it's not enough to be considered "proper" research, distinct from securing funding.
> The process is completely broken.
You're not going to find a lot of disagreement that the process is broken. But one thing I've often found is that those who are part of the process and those who are outside of it think it's broken in different ways. Sometimes people outside of the process suggest to fix it in ways that will make things worse or are not actually a problem. Sometimes the solution proposed would in fact likely cause more problems that those outside the process can't anticipate, or were fixes to problems that have already been patched.
It's a lot like systems engineering -- when you go to a new codebase and start changing legacy code that looks wrong because you know a better way of doing it. Well... maybe it's not wrong but that way for a reason, and you just don't know it yet.
> Your colleague is right that being a PI in a research lab is a lot like being a startup founder. But we shouldn't take that to mean that research labs should be startups, with goals like establishing a profitable revenue stream that will assure a self-sustaining operation. That's just not the purpose of research, and trying to make it as such would change the kind of research being done.
I agree that research should not be about creating a profitable business stream. This is not what I meant to say. I meant that this creates a constant pressure to find funding, you essentially can never switch off, and you never get to a stage where you can say, ok now I can focus only on research for a while. It also creates an absurd incentive of not taking risks, because if things don't work out you are likely not getting funding again because of the gap in your research output.
> The other thing is that like a startup, research labs get a significant chunk of "startup capital" in order to get their research agenda off the ground. Depending on the agenda, this can range from 10s of thousands to a million dollars or more.
This is highly dependent on the country and university, but it should also be said that even a 1M startup does not last very long if you are an experimentalist (it often just barely pays to equip your lab with the essentials).
> Third, there is other institutional funding that goes beyond that. It's true that most funded research is through grants, but at the same time it's hard to get grants without some prior research results. That's where institutional funding comes in -- to give you a little boost to bootstrap some results that you can then take to a funding agency.
I think this is were experiences will differ dramatically between countries. For example in Sweden academics have to find ~50% of their own salary (+80% overhead), some of this can come through extra teaching or adminstrative duties. In Germany a professor in engineering will typically get 1-2 PhD student salaries with their position. What I know is typically done is that you apply for a grant were you have done significant work already (also helps with the feasibility) and then use that grant to do research to apply for the next grant. Institutional funding plays a very minor role typically.
> So "researchers" in research institutions spend most of their time looking for funding.
>I think I know why you put "researchers" in scare quotes, and that is because of a common perception that "research" is limited to the technical work of running an experiment and writing papers (please correct me if I'm wrong). But I disagree with this view, and instead I consider research as also including getting the money.
Actually this was not meant to be scare quotes, I put the quotes because researchers is such a broad term (there are a researchers in industry or the big national research labs, where much of what I said does not apply to the same degree).
> Why? It has to, because getting the money involves setting clear research objectives, communicating those objectives to others in an understandable way, forging alliances with other researchers, navigating regulatory obstacles, hiring competent research personnel, designing research experiments, and most importantly managing the project over a decade or more without the whole thing imploding.
I don't know where you are but most research grants projects I'm aware of don't last for 10 years (5 is already considered long, 3 is much more common for ones I'm aware of). That said, I think the list of responsibilities is crazy, because generally you can not hire anyone to help out with any of those, and there are the administrative and teaching duties that are on top of that. No wonder that most academics work well over 60h per week.
>That is research, because if you don't do those things you're just doing a "project". Getting something into the world and getting buy-in from external agencies is important precisely because there is no revenue stream and no customers who provide feedback to your venture.
See and there I disagree. The above responsibilities actually lead to the least risk taking possible. If you constantly need to convince people that what you're doing is great, based on somewhat arbitrary metrics which typically do not reflect the actual importance of the research, then you get academics optimising for short term metrics.
>Sitting down and doing the coding, running experiments, or even writing the paper is important to the scientific research process, but it's still just a part of it. Certainly it's not enough to be considered "proper" research, distinct from securing funding.
I agree, project management designing a research project etc. are important parts of aspects of research. However, getting research funding should be a means to an end. What I often observe is that obtaining research funding becomes the end goal, largely because of necessity.
>> The process is completely broken.
> You're not going to find a lot of disagreement that the process is broken. But one thing I've often found is that those who are part of the process and those who are outside of it think it's broken in different ways. Sometimes people outside of the process suggest to fix it in ways that will make things worse or are not actually a problem. Sometimes the solution proposed would in fact likely cause more problems that those outside the process can't anticipate, or were fixes to problems that have already been patched.
I should clarify, I'm writing this from the inside I am a research academic, have attracted significant funding etc.. I also agree that outside solutions are often very much missing the mark, in particular solutions from communities like HN who often look for quick technical solutions for social problems.
>It's a lot like systems engineering -- when you go to a new codebase and start changing legacy code that looks wrong because you know a better way of doing it. Well... maybe it's not wrong but that way for a reason, and you just don't know it yet.
The issue I have is that we have seen in fact seen strong changes in research funding over the last decades. The ratio of competitive vs non-competitive (institutional...) funding has increased strongly in most western countries I'm aware of. At the same time reporting responsibilities and the importance of metrics has become bigger as well. All this based on the reasoning that there needs to be accountability for the money the government spend on research. I think we are seeing the failures of these changes now.
I think the root problem here is not that money is needed to do research, but that negative results don't advance an academic career and actually hinder the ability to get future monies. It's a very hard landing if you take too big of a risk.
Honestly I think needing to constantly get external approval for one's research is a very helpful forcing function for academics, who in my experience can be quite... nose down in their work to put it mildly. How can a researcher evaluate the actual importance of their research without larger buy-in from the public? Certainly many think whatever they are working on is of the utmost importance for the world, and they all have quite a long story to tell you how.
> I also agree that outside solutions are often very much missing the mark, in particular solutions from communities like HN who often look for quick technical solutions for social problems.
That's exactly what I was reading into your words, my apologies.
> All this based on the reasoning that there needs to be accountability for the money the government spend on research.
But what's the alternative? We all agree research takes money, and money is limited. We have more research to do than we have money to spend on research. Therefore we can't research everything, and must instead choose how to allocate our limited funds. I suppose at this point we could say that research funds are allocated by a lottery where no one proposes anything. Some people feel the current funding model is akin to a lottery. But I don't think we really live in a world where allocating lots of public money to unseen projects is going to fly. The only alternative then seems to be that researchers should put forth a description of their proposed research agenda so we can discern one project from the next...
You see where I'm going with this? Every attempt I've seen to revamp the funding process inevitably reinvents it, or tries to turn it into a corporate profit-seeking model. We already have that, so I don't think we need to duplicate it in another context.
I'm not sure why the author is convinced this approach won't work, but there is certainly a better option for any country that wants to address the situation, and it's been done in many other areas. Establish federal funding agencies devoted to the goal, and use those funds to establish dedicated research departments at leading universities, which employ professors and technicians who bring in grad students and postdocs to work in their labs. The result has been very successful in areas like cancer research - childhood leukemia has gone from a almost uniformly fatal disease to one with a 90% survival rate. Why? How many universities have some kind of NIH-funded 'Center for Cancer Research'?
However, you won't find any such comparably funded centers for renewable energy research at leading universities, because the US government won't fund them through the obvious agency, the DOE. DOE budgets make for interesting reading, because it becomes clear that the history of the DOE - from Manhattan Project to Atomic Energy Commission to 'DOE' - rather reveals it was set up mainly to deal with the nuclear weapons program, and the vast amount of highly toxic waste it generated. The DOE foray into 'clean coal' initiated around 2000 has been one of the most fraudulent 'scientific' programs ever run in the USA, from bogus nonsense like 'FutureGen' on down. They were characterized by 'public-private partnerships' which allowed DOE partners (coal and oil corporations) to hide the data about their gross failures behind proprietary claims. Literally billions of taxpayer dollars have been poured into these rat holes with nothing to show for it for over 20 years. Case in point:
https://en.wikipedia.org/wiki/FutureGen
This leaves countries who have invested heavily in basic R&D in the solar/wind/storage sector well ahead of the USA. China is certainly the leader in solar, having mastered large-scale production of monocrystalline silicon wafers for high-efficiency, durable PV. Germany and Japan are in a similar position. Australia, even with its heavy reliance on coal, has long funded innovative solar research. The US government instead has devoted itself to funding fraudulent fossil fuel games (whatever CO2 gets captured at DOE-financed plants has almost universally been used for enhanced oilfield recovery, and much of it just gets boiled back off to the atmosphere during crude oil processing).
I don't know why the author didn't mention any of this, but if you're going to base an article on a line about Musk's $100 million effort, you might want to dig into that history, right? As far as Musk's approach, here's the latest update on how that's going:
https://grist.org/technology/elon-musk-funded-carbon-removal...
However, if the USA really wanted to get on top of this and move towards a rapid transition to renewable energy (read: 3% reduction in fossil fuel production and consumption per year, with an equivalent growth in renewable energy to meet demand), it would probably need a new funding agency outside the DOE/NNSA, with a budget comparable to that of NIH, i.e. $30 billion a year. Given that the USA is almost as much of a petro-state, economically speaking, as Russia and Venezuela are, I don't really expect much progress on this from our fearful political 'leaders' (who are, in reality, more like corporate middle managers).
From the criteria: "[...] and that they have a pathway to scaling up to 1 million metric tons removed per year. (The most conservative estimates say the world will eventually need to remove 1.5 to 3 billion metric tons of carbon per year.)"
So 100 MegaUSD for someone to deliver something with the only the potential of being a 1 promille solution: not sure if this strikes me as super promising.
This approach seems interesting: > "Heirloom, a company working on capturing CO2 directly from the air, recently made headlines after raising $53 million in venture capital. For the XPRIZE, Heirloom teamed up with a company called Carbfix, which specializes in turning CO2 into stone. Carbfix is already demonstrating its technology at a working carbon removal facility in Iceland."
Note also that there's a real tie-in with SpaceX here, which runs rockets on a variety of fuels, i.e. RP1 which is essentially highly refined jet fuel, and methane, CH4. Mastering carbon capture and conversion technology, i.e. running processes like CO2(atm) + H20 -> C + H -> CH4, long chain hydrocarbons (RP1), would be critical to synthesizing fuel on Mars for example, which has an atmosphere of CO2 and apparently, water ice in the subsurface. This could also make SpaceX independent of fossil fuels: imagine a giga-scale carbon capture and methane synthesis factory in Texas providing all the fuel Starship needs.
Also, in a fossil-fuel free world, international air travel would require jet fuel, and that's possible with DAC and water-hydrogen, via a modified Fischer-Tropsch process, first used with coal gas / steam as inputs.
The real issue with DAC is that it is incredibly difficult to innovate around the fact that CO2 in air is just immensely dilute. You need to process enormous amounts of air to remove an appreciable amount of CO2 and, even worse, as the plant operates and recirculates processed air, the local air around the plant becomes more and more devoid of CO2 leading to a decreased amount of CO2 captured per unit volume of air.
The only real improvement that I could see occurring in this space is coming up with a process that creates less back-pressure against the fans pumping in unprocessed air which could bring down the energy cost per ton of CO2 removed. But even in that case the back pressure created by a DAC process is normally caused by flowing air through a porous catalyst, which is essential for high efficiency. So there's a trade-off there as well.
Ultimately I am not very sanguine about DAC and I have been disappointed to see news agencies reporting as if DAC is even in the top 10 technologies most important to reduce carbon emissions.
Proof-of-process for Starship fueling would be generating a stream of pure CO2 from air at a rate sufficient to feed into a methane production facility (which would require a similar stream of hydrogen from water processing) to produce around 500 tons of liquid methane in a reasonable amount of time. Possible?
See figures SPM.5 and SPM.7 of the "Mitigation of Climate Change" IPCC report for a picture of how much needs to be cut to reach a stable state, and how much Carbon Capture is estimated to contribute.