Comment by fiprofessor | Hacker News Reader

Comment by fiprofessor | Hacker News Reader

Parent Full thread

I see a lot of appeals to "fund people, not projects", but the main issue I have with it is that I think it strongly biases people to select "super stars" who have impressive credentials / come from high status organizations. Of course, there's already a lot of "rich get richer" effects in science, as the article points out, but I think it would be nice to try to shift away from those things, not increase them. (Besides, isn't the resounding message in academia these days supposed to be that assessing people is full of biases that we ought to avoid? Of course you always assess the people behind a grant application to some extent, but focusing on the project would seem to be better in terms of avoiding these biases.)

The other alternative of "funding lotteries" is more suspect. As the article summarizes, the argument for lotteries is:

> Advocates of lotteries make two key critiques: a) the current system forces researchers to spend a lot of time preparing grants; and b) peer reviewers cannot reliably identify “good” grant applications. They claim that a lottery system would reduce the time spent on review (because reviewers would mostly skim the proposals to check for minimal scientific robustness) as well as the time spent on preparing proposals (because there would be less of an incentive to meticulously craft proposals, given that no matter how detailed and well written they are, they are going to be chosen at random).

Of course (a) is true, but as the article suggests, the evidence for (b) is a bit shaky. Moreover, we should have strong priors against (b). To be sure, reviewing is not perfect, far from it! But, scientists must at some level be able to judge the future prospects of work, or else they wouldn't be able to make any fruitful decisions about what research to conduct. So the primary plausible way I see that (b) could perhaps be true for a given pool of applications is if all of the ones in the pool pass some thresh-hold bar for quality that makes it hard to further distinguish between them. It's not clear to me that that quality level would be maintained if we move to a lottery.

I see a lot of appeals to "fund people, not projects", but the main issue I have with it is that I think it strongly biases people to select "super stars" who have impressive credentials / come from high status organizations.

Please, someone, correct me if I'm wrong, but I think this is essentially how the European Commission allocates scientific funding--professors basically get big block grants to fund their labs to do whatever. Then there are some big grants that go to institutions, with those like ETH Zurich soaking up a lot of the funds, and then those institutions can distribute to the smaller ones as they see fit. Rubbing up to high-profile professors at ETH or GeoForschungsZentrum feels a lot like petitioning King Frederic of Prussia, rather than anything merit-based.

It's really not clear to me that this is a better system; at least in my field (geosciences) I don't think that the European labs are more innovative or productive than their American counterparts, which is what the 'fund people, not projects' idea is supposed to enable. Furthermore, it completely blocks nonacademic research funding, or much project-specific funding. For example, I work for an applied science nonprofit based in Italy, and we have a lot of ideas that would (in principle) be big hits at NSF, but there is no real avenue to fund ~€100-300k projects on an ad-hoc/project-specific basis.

> but there is no real avenue to fund ~€100-300k projects on an ad-hoc/project-specific basis.

Long time ago I had a colleague who lamented that his projects were never selected by the R&D funding committee of our employer. I suggested him to multiply by ten his proposed budget and then at the next round his proposal was accepted! This was at a time were there where much more money in the Telecom sector than now.

So please do not hesitate to multiply by 10 your proposed budget. It does not have to be more ambitious on your side, all is needed is to find additional partners who could bring value to your proposal, for example by implementing it.

(I was both work package and project leader for EU FP7 projects)

How does it benefit the EU tax payer when projects that could be successfully done for 300k, are blown up to 3M with questionable added value? That's exactly the kind of problem that needs to be solved: You are just advocating to game the EU bureaucracy better.

> How does it benefit the EU tax payer when projects that could be successfully done for 300k, are blown up to 3M with questionable added value?

At least in FP7, the EU wanted implementations and new markets or businesses, because they didn't want to pay for something they didn't understand.

Trade metrics are easy to understand and less objectionable than funding someone with an innovative idea on a domain where there are few experts. Th EU commission themselves has also to report about their actions to EU parliament and EU council.

> You are just advocating to game the EU bureaucracy better.

You may not know but it happens that EU commission, or national authorities ask for reimbursement of funds if they thinks they have been gamed. Tricking the "EU" is not advisable.

You are just restating that for the EU, bureaucracy is more important than genuine research. Furthermore, the "gaming" I referred to is of course the legally allowed game that the EU is actually expecting you to play, and that you advised in your previous comment.

> You are just restating that for the EU, bureaucracy is more important than genuine research.

Perhaps but that's not my impression. My impression is (as in this thread) there is no good intrinsic metrics for research, except if it shows an impact on society.

There is no good intrinsic metrics for research, we can agree on that. Impact on society is not a good measure either, because if you had funded 10 smaller research projects instead of one big one, you might have gotten a much bigger impact on society, but not so easy to measure and attribute, and maybe not as quick to do the impacting.

So yes, bureaucracy. Because instead of actually doing the job that needs to be done, the EU is more interested in covering their own asses so that it looks as if they were doing the job. But the EU is not special in any way here, except that it is a very big example of a bureaucracy, of course.

This reminds me of many anecdotal reports of companies having difficulty selling in Japan, only to be told to "add a zero" to their prices and find huge success. Too cheap = not very good/desirable, in other words.

> Too cheap = not very good/desirable, in other words.

I think that's the EU mindset (at least in FP7), it wants their funds having the greatest impact as possible, certainly a small project can't have much impact.

I think funding lotteries are rarely proposed as straightforward lotteries, there is usually still an evaluation step that filters some 50% of proposals. It’s based on the observation that there is usually consensus about what is a reasonable grant proposal, and very little consensus about what is a great grant proposal.

Yes, I agree that it's the fine distinctions at the upper end that are hard to make. But if we're going to eliminate 50% of the proposals, is that really going to reduce the amount of time people spend on them? It might reduce reviewer time (which would certainly be welcome!), but I'm not so sure it reduces the submitter's time.

It comes down to whether the large amount of time spent is from people polishing and re-submitting stuff that's good but not "great", or whether it comes from the initial stages of getting something to that good quality to start. In my experience, it's the latter. And if that's typical, then I don't see how the lottery helps.

> But if we're going to eliminate 50% of the proposals, is that really going to reduce the amount of time people spend on them?

It should do - if you're only aiming for the top 50% of proposals rather than aiming to be the best, then you can submit a proposal that's "good enough" rather than putting in a lot of time squeezing out the final few percent of marginal improvement.

If it’s a lottery you don’t need to make a hard cut off of 50%. You just pass all proposals that meet the criteria. That means you only have to design the proposal so it meets those criteria and any extra work on it is useless since it will be ignored and just get thrown in the same pile to be randomly chosen.

What happens when, and I know this will be rare, our ideas of reasonable are wrong? Seems like that ends up being a dead end where the research needed to correct our misunderstanding can't get funded.

In Ricón's Fund People Not Projects III article [0] he relates:

> Ioannidis et al. (2014) note there are 15M scientists that published anything in the 1996-2011 period, but only 1% that has published every single year in this period. This smaller 150k-strong group accounts for 40% of all papers and 87% of all papers with >1000 citations.

One percent seems like a good size for a major niche; those who do the bulk of the publishing.

This is important, but if that's the only niche you're looking at you're bound to miss 99 percent of what you want to find.

[0] https://nintil.com/newton-hypothesis

Edit: Maybe the people that need to be funded most are secretaries for some of the other 99 percent.

> you're looking at you're bound to miss 99 percent of what you want to find.

Or those 1% are real professional that know their stuff while the other are publishing because they are doing some internship, or publishing low quality articles on an obscure domain is a way to evade criticism (how many ALS experts in academia?) or other short commitment to a topic.

I read a lot of literature on ALS (Lou Gehrig's disease) and clearly 99% of publications have no value at all, even when their university claims of "a breakthrough".

As far I know there were breakthroughs only every ~5 years (TDP-43 in 2006, C9orf72 in 2011, FUS 2010) and no similar breakthrough since. Big companies like Biogen, which bet much on various genetic (ASO) therapies, are now removing drugs from their pipeline. We clearly do not understand the field, despite publishing 15,000 articles per year!

The main issue with the people-not-projects idea is a transition problem. New systems are desired because the current system isn't working well enough. How do you identify the superstars? By relying on metrics created by the current system.

In particular, any system that identifies superstars would probably have identified Lesné, the guy who seems to have been forging Alzheimer's research. He had already obtained huge and lavish grants for his lab on the back of this work. Winner-takes-all is a system that would throw accelerants on the fire of fraud, as presumably once a scientist has managed to get a blank cheque cut, nobody is incentivized to scrutinize how that money is being spent too closely. They're a superstar, after all.

I think that a lottery has potential if we use peer review as the bar for quality.

My prior is that peer review is better than random chance. This seems sensible; after all, a human being should be better at determining scientific validity versus random chance. Under that assumption, it makes sense to have peer review to narrow the pool down to a smaller number of grant proposals.

It doesn't seem unreasonable that peer review could get the "best" proposal into a pool of 10-15 applicants around 90% of the time. I think that would reduce the pressure on proposals by a decent amount.

Isn’t that what’s broken? Everyone is trying to game peer review.

there are so many major problems with 'fund people not projects'. Science is already a little bit broken. Scientists aren't supposed to dragnet data until they get the results they want, but it still does happen. however, if we were funding people and not projects, this would happen almost every time.

I also think politics in science would become even more of a problem. all of our money towards science would go toward funding research that backs their worldview.

I also think we'd see even more social media like driven science. "I am going to research this because I can write on social media/youtube on this and get lots of click/likes-> money"

Reply on news.ycombinator.com