The PhD Metagame: Don't try to reform science – not yet
maxwellforbes.com
maxwellforbes.com
Power makes people stupid: powerful people can't imagine a world other than the one that brought them their power. They will say, "That's the way the world is." Let's encourage students to continue to imagine other possible worlds in order to challenge the status quo.
The concept of lobbying itself has been basically shattered in our modern world with businesses having a near infinite amount of resources to exploit it. I don't think there's anything implicitly unreasonable about conveying your understanding of the importance, impact, and potential consequences of major choices onto key decision makers.
They tend to be intensely practical and specific, rather than hot morally heated topics. Like building infrastructure, securing a government contract, or amending/removing a new regulation from your sector of business (e.g. making sure a new law on tobacco exempts cigar manufacturers).
I'm from that community, was there when they presented it, have used and still use BERT a lot, and still if it were my decision I wouldn't have given it the best paper award, even in hindsight.
BERT the model has been, of course, enormously influential. I still use it, even after the generative LLM revolution (which also stands on it shoulders). I greatly respect its authors and am truly grateful that they published and open-sourced it.
BERT the paper? It's not really well-written (almost everyone who wants to understand Transformers or BERT turn to a blog, because the papers are so bad) and it's not stellar in terms of scientific insights, because indeed, it scales some stuff up and comes up with a lot of magic numbers that are there presumably because other alternatives were tried and those happened to work. Or maybe not even that, because some stuff included in BERT actually turned up being useless (see RoBERTa), so I guess they just winged much of it and it worked.
From a scientific paper, I would expect much more explanation: why the architecture is designed like this, why this number of layers/dimensions and not that, etc. which that paper thoroughly lacks. No one will learn to do better science from that paper, it's not a paper that a PhD student would benefit much from reading except as a curiosity (they of course can benefit from downloading, using and getting to know the model. Just not from the paper).
Maybe create a "best model award", "best software award" or whatever, but in my view as an academic a best paper award is just not for this.
No one really cares how well written a paper is.
Writing quality is often a make or break thing when it comes to whether a paper is accepted or not, mostly because it makes a paper easier to understand, ie its contributions and the evidence that they are truly there are easy for a reviewer to pick up on and appreciate.
Furthermore a well written paper is a far larger contribution to the research community - the audience post reviewers - than a poorly written paper with the same contributions, for the same reasons: well written papers can be a joy to read, particularly if a leap in contributions are presented in an easy to understand way.
I understand that these things are cultural (ie field specific) but this has been my experience.
A well written paper transports the reader to the perspective of the writer. It’s not about poetry or aesthetics.
Bad handwriting is illegible. Good handwriting is clear. Calligraphic aesthetics is another domain.
The process of experimentation is what makes Computer Science "science"!
- which other options you also tried, but which failed
- write down hypotheses that explain the worse results of the other options
- write down hypotheses why the chosen options gives better results
- ideally formulate testable predictions that these hypotheses imply
- etc.
Simply saying "other alternatives were tried and those happened to work" is not science, but tinkering around combined with magical thinking.
I don't know how you'd improve on the former. For a lot of it there simply isn't any sound theoretical foundation, so you just end up with flimsy post-hoc rationalizations.
While I agree that it's unfortunate that people often just present magic numbers without explaining where they come from, in my experience providing documentation for how one arrives at these often enough gets punished because it draws more attention to them. That is, reviewers will e.g. complain about preliminary experiments, asking for theoretical analysis or question why only certain variants were tried, whereas magic numbers are just kind of accepted.
I'd say that's a bit strict take on science, one could be generous and compare it to biologist going out into the forest and combing bsck with a report on finding a new lichen.
Thought admittedly these days the biologist is probably expected to report details about their search strategy, which the sticky-wall researchers don't.
Rather that when you do come to something empirically, you need to validate your findings by e.g. ablations, hypothesis testing, case studies, etc...
So great science would come up with a sound theoretical foundation, or at least strong arguments as to why no such foundation can exist.
> A new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die and a new generation grows up that is familiar with it … An important scientific innovation rarely makes its way by gradually winning over and converting its opponents: it rarely happens that Saul becomes Paul. What does happen is that its opponents gradually die out, and that the growing generation is familiarized with the ideas from the beginning: another instance of the fact that the future lies with the youth.
https://www.benwhite.com/misc/good-ideas-need-to-outlive-the...
Please don't turn this -- or any hypothesis -- into a "law" of how science works.
If the existing theory predicts results that are 50% off, and the new theory is 45% off in the other direction, then things aren't likely to be accepted. If it is instead 0.1% off, that makes a much stronger argument. The issue with rejecting the first case outright is that often the experiments themselves have imperfections, but those are much slower to work out and refine. When the new theory doesn't need the existing experiments to be refined, I would guess long term experts are much more likely to entertain it.
Might I venture the guess that seeing tangible advantages for oneself when using that new theory is more important? If you adopt it, will you become one or more of more famous, more successful, able to advance your own work, become part of a more reputable group, etc.?
If it is merely something that will not affect you, there is little or no incentive to change one's view.
But scientific truths have a tendency to be adopted that falsehoods don't because truths are verified by other truths, whereas falsehoods contradict them. A lie can only be so large before it becomes self-contradictory, the truth is incomprehensibly large yet coherent.
If quantum mechanics has problems, as we do more experiments, we'll encounter these problems more and more, until eventually they can no longer be overshadowed. I predict quantum mechanics will never be entirely thrown out, but it will end up as a simplified approximation of a more complex "true" model; which is still taught in schools and used wherever extra accuracy isn't needed, like classical mechanics is today.
I think Max Planck's quote matters in practice too. In theory, you can discover a "scientific truth" and be recognized for it, but only long after your death, and only after someone re-discovered it (i.e. you didn't advance scientific knowledge at all).
However, most people aren't particularly unique, which means that if you discover something, chances are others have discovered it, or are at least close enough that you can easily convince them. You may not be able to convince the dominant "in-group", but if your idea is obvious enough (which it probably is if it's true and you managed to discover it), you can form an "out-group", which will grow as the idea gets verified (by truths) far more often than contradicted (by lies, because there are less of the latter, since a lie can only be so large without being self-contradictory).
Why do science in the first place? If your only goal is to predict something, you're doing it for yourself, so do "Science 1" and listen to others, but only to correct yourself. If your only goal is status, the truth doesn't matter to you, so do "Science 2" and make others happy. If your goal is to further scientific knowledge, I recommend you do both with preference for "Science 1": prioritize being correct, but explain your idea very well and make others happy when possible without sacrificing correctness (diplomacy).
It's important to note that when you can't change the majority's factual belief, you should really evaluate your own, because usually in such cases you're the one whose wrong. But otherwise: when you can't change others beliefs, the next best thing is to (as much as possible) not care what they believe, even if they are the majority.
This is non sequitur. If you do more experiments, you'll encounter it. But if there were such a wrong theory that everyone insisted was correct, it's not necessarily the case that more experiments will occur. I can't do an experiment to invalidate... quantum mechanics is in a regime where anything less than hundreds of thousands of dollars does not even get you started (never mind the lacking expertise). It's also unlikely to be financially lucrative in the timespans that would entice an investor (especially one averse to pissing off the status quo, as most are). By doling out the grant money (or not) to those people who will preserve the status quo, one could let the false theory survive for decades or even centuries.
We seem to have this mythology of science from a past era, where some maverick can just bust in and start embarrassing people with unignorable truths. If ever there was such a time it exists no more. The stakes have never been lower, trillions won't be lost if we get quantum mechanics wrong. Thousands won't die in a quantum mechanics accident that could have been avoided. The problems could persist indefinitely.
However, right now people are spending large amounts of money to build quantum chips. They're not explicitly trying to disprove quantum physics, but implicitly testing it through their experiments. And if these tests suggest that quantum physics has fundamental issues, they'll investigate, at least so if they realize quantum chips are impossible, they stop spending money trying to build them.
The problem is rather that a lot of physicists tend to "hand-wave away" that existing problems (like "what is a measurement?" and "sudden collapse of the wave function") actually are problems in the theory, i.e. that we don't have "the theory is basically correct (as evidenced by lots of experiments that were done), but these 'problems' are simply open, unanswered research questions".
This is exactly an example of the "hand-waving problems away" point that I made. :-)
I'm an industrial physicist. I've noticed that physicists, and the general public, often have different ideas about what problems we should be versed in. And people are surprised when they learn that most physicists are not theoreticians. Were it up to the public, we'd all be working on warp drive, infinite energy, and explaining quantum mechanics. ;-)
We all learned about the problems in both undergraduate and graduate training, and in discussions and readings. I attended a lecture about it by John Bell. I was excited about it, but I also had an experiment to finish.
I think physics is utterly unique in having a theory with seemingly infallible predictive power and zero explanatory power. But if someone asks me about it in the lunch room, all I can do is shrug it off. The fact that this paradox hasn't stopped us dead in our tracks, in 100 years, that's the problem.
Rather say: you have studied physics, but what you actually work on and are interested in is engineering. :-)
Addendum: Just to be very clear: there is nothing wrong with being exciting about engineering problems from industry.
https://en.wikipedia.org/wiki/The_Structure_of_Scientific_Re...
Almost every part of society, including science tends to collapse towards those that have access to ideas and resources, and are placed within society to be able to take advantage of them, and those that are exploited. Even the Nobel Prize was found to mostly favor those who were born to win a Nobel Prize. Best way to win a Nobel Prize? Be born to a wealthy family with a prestigious science background.
Those that have access tend to devolve into monopolistic tendencies, exploiting the existing "paradigm" for their own gain, while punishing, excluding, or minimizing those that suggest alternatives that might in some way disrupt their power structure and control. Money, politics, military, industry, and a lot of other parts of society seem to all work about the same. Best way to work at the White House? Be born to a wealthy family with political access.
Almost every person who has retrospectively been considered "great" or "revolutionary" centuries or millennia later were punished, excluded, or minimized during their own lifetime, while their contributions were then reevaluated, fought over, fetishized, and collected after their death. Galileo was mentioned elsewhere, relatively well known example, some success and acceptance at the time, mostly condemnation for "vehement suspicion of heresy." Van Gogh's work only began to attract attention in the last year of his life ... right before he shot himself in the chest with a revolver from being so mentally tortured.
Someone’s plan when investing in early Solar panel R&D went something like: If everyone would just… “follow their economic interests” driving down the cost of panels will dramatically increase adoption further driving down costs in a feedback loop.
Unlike most “if everyone would just” plans that one actually worked because the desired behavior aligned with people’s interests.
I'm not sure if that is a good paralel. The difference is that we didn't needed "everyone" to innovate on solar panels. It was enough if "someone" was, and those who did not got left behind with their inefficient processes. That's not a true "everyone would just" situation.
The customer base continually expanding is the “tough” side of the equation. 100’s of millions of people behaving differently is the hard part of those “If everyone would just” plans.
Its OP. Signals engagement and understanding to society, while expending no energy on work. Idealisation is not retardation, its optimisation.
> “If everyone would just...” then you do not have a solution
What I mean by "not be a dick" is to check the alignment, the goals of the process compared to what we're actually achieving. What is the point? The author of the article lays out a lot of reasons and even is stating how these things are well known. Which unfortunately means someone needs to actually take action. When we're in a situation where many people want change but no one is willing to fight for that change, then we will just keep doing what we've been doing and headed where we've been headed. Even if that is knowingly off a cliff.
I don't need everyone to just do something, I only need a few more people to stand up. And yes, I will tell those that are saying "keep your head down" to shut up. Some things are worth fighting for and for me, one of those things is the integrity of science.
>I don't need everyone to just do something.
Naked contradiction. Either everyone needs to just do their job or not everyone needs to just do their job.
>Surely "do your job" has to be a reasonable version of this.
There are entire fields of research centered around answering why people don't 'just' do their jobs like good little worker bees in exquisite detail. Some terms I'm aware of that you may find useful to look into, in rough order of how general to the problem they are: Agency problems; the Case theorem; malicious compliance; work to rule; collective bargaining; moral hazard; perverse incentives; adverse selection; rent seeking; regulatory capture. If you want to read up on people trying to design actually working systems from scratch, look into the world of mechanism design, starting with auctions and branching outwards.
>When we're in a situation where many people want change but no one is willing to fight for that change, then we will just keep doing what we've been doing [...]
One could argue that the past ~century of scientific and technological development has probably beat any other 100 year period your could pick hands down along any natural metric. So "what we've been doing" is actually pretty great, and it may not be a good idea to stake such a hugely important enterprise on some newfangled and only theoretical ways of doing things.
> >I don't need everyone to just do something.
> Naked contradiction. Either everyone needs to just do their job or not everyone needs to just do their job.
It's not a contradiction; "just something" is not the same thing as "their job". "I need everyone to do their job" does not contradict "I don't need everyone to just do something." (Emphasis added for clarity about the differences.)
> One could argue that the past ~century of scientific and technological development has probably beat any other 100 year period
My reply is the same to the other arguments you've made
Many of those authorities have learnt that deciding what is true based on reality is good and all, but you live longer and better by making friends and not disagreeing with them.
This is a societal failing.
I would characterize the problem with science as being a failure to increase the available resources commensurate with the population of people capable of doing science. In this situation, the competition becomes sufficiently fierce that it is statistically better to lie, cheat or knee-cap your competitors in some other way than it is to actually do good science, which is unreliable. What you see as fealty to scientific authority is actually just a system which has become totally dominated by resource competition to the exclusion of its actual purpose.
That's not inevitable. I count myself lucky that it didn't happen to me.
Unfortunately I don't know how to improve schools to the level of those I attended over half a century ago. And I lack the get up and go to make it happen anyway.
> Almost no one is a dick on purpose.
What is important is to have self reflection and to recognize when you have been a dick, to apologize (make amends if necessary), and try not to repeat. Yes, habits die hard, but we can still improve. But the biggest dick move is to double down. We've created a culture where we act as if being unintentionally wrong is a bad thing and that the worst thing you can do is admit a mistake and self-correct. But we are always wrong (to some degree) and so the only thing there is is to self-correct.
So yes, we can stop being dicks. It's how humans evolved. We wouldn't have expanded from our small tribes to villages, to cities, to countries, to a global economy if we weren't capable of this. The arc may be slow and noisy, but it has always expanded to be more inclusive.
Humans have always advanced through the formation of coalitions. To optimize your own success you have to simultaneously optimize the success of others.
In fact, this is even how I review papers. I am much more detailed than my peers, and get very specific. I always also include a list at the end detailing what factors are the most important and what I think the authors could do to change my mind (if I'm rejecting). If I'm accepting, I'll also argue my points to the other reviewers and make them stand for their arguments.
Truthfully, if no one is willing to change their minds, I'd say they can't be a scientist. It is a fundamental requirement simply because we are all wrong and all the time. While we can get ever and ever closer to it, absolute truth is fundamentally unobtainable. So you must always be able to update your beliefs, or else you will become more wrong as time marches on.
[0] I also recognize that the inability to convince me does not mean I am right and the other person is wrong. But this too is why I specifically make a point to try to help the other person. At least as long as I believe they are acting in good faith. If I am wrong then I WANT to know. I take no shame in being wrong, but I take a lot of shame in being unwilling to right myself.
If your goal is to seek truth, then you need to reframe the setting. It is not "I defend my position and they make their case", that is allowing yourself to change but framed to maintain your current belief. Sure, you have good reason to maintain your belief and I'm not saying you shouldn't hold this, but it should be a byproduct of seeking truth rather than the premise.
Pick anything. Climate change is a big one. I would definitely have to eat some chaff if it was shown to be false personally.
Groups display some sad behaviours where most of the members know that what they are doing but the leadership is committed to something silly so they go along anyway. Sometimes it gets bad enough that they quit the group, but that is all they can do since most never have a chance at reforming it. This turns up formally in corporations (the executive team have the power to reform) but that is actually just a mirror of a natural social dynamic.
Consider religion for the more natural form of this. It gives a good indication of how the ratio of insider to outsiders pans out in practice. Catholicism is a very well established one, and how many people have the moral and practical authority to reform their doctrine? Not many.
In fact university systems are practically religions unto themselves, with perceptible and distinctive campus cultures, Greek fraternity/sorority and what have you clubs to join, there is certainly a churchy blueprint being followed in education.
It seems to me that religion and medicine and architecture and other various sciences were tightly intertwined. It's why the reactionary Christians are rejecting Science qua Science and Scientism, because if science is not subject to religious ethics or morals, then science has power that church doesn't.
I don't know about Asia and the rest of the world, but seeing as the Catholic Church set up systems such as universities and hospitals, it's altogether unsurprising that religion is woven into the DNA of those disciplines even as they secularize and syncretize.
There's no schism. Religion and science are both ostensibly attempting to describe the underlying truth. The issue is that any institution/engine that has money/power, will obviously attract people that want money and power too, truth be damned.
The idea that the institution is about 'uncovering truth' is certainly given lip service/held as an article of faith as this is required by the naive masses to invest themselves in to the idea (science/religion), but the underlying reality is a somewhat dirtier political jostling for money/power.
I am unfamiliar with this term, and it gives me pause. If an organization does not seek truth, what does it seek? Also, religions typically consider themselves as guardians and authorities of Truth, and disseminate/preach it through leadership/missionary/evangelization activities. Surely, the seekers of truth are ones who join and follow such paths?
In terms of Christianity, "finding the Truth" involves following Jesus and sacrificing our lives to do it. Sure, a televangelist or notable preacher could line his pockets, electroplate his Learjet, and influence politicians, but a follower's "profit" is moral and intangible, in exchange for actual cash, goods, and services we donate freely, or that's not tax-deductible!
In terms of universities and science, I'm curious how you became educated in Number Theory without joining a truth-seeking university, picking up truth-seeking textbooks, and hearkening to truth-seeking professors. Science as an industry or career path attracts ordinary people trying to make a living, whether that's in a community college, a four-year, or a prestigious Ivy League or Oxford/Cambridge setting. Certainly, the money, power, and prestige are attractive and draw in followers just the same.
It’s weird how people talk about our modern very unnatural corporations and how they behave and then rush to insist that it’s all a reflection of totally natural and organic stuff.
Yeah I think they doth insist too much.
>Naively, you’d think such a revolutionary paper would be met with open arms. But when it was given the best paper award (at NAACL 2019), the postdocs I talked to universally grumbled about it. Why? It wasn’t interesting, they bemoaned. “It just scaled some stuff up.”
I still hear people making this complaint, despite the extraordinary success of scaling over the last few years.
It's pretty clear that scaling is the winning method (although exactly how to make best use of scale is an open question), but many researchers find it repulsive.
"Winning" in the sense of "perhaps suitable to build practical applications that make a lot of money" - perhaps (even though I'd claim that whether this is true is still an open question).
On the other hand, "winning" in the sense of "getting a deeper understanding why the method works and having a model that can be analyzed deeply very well", then I would clearly say that scaling is not the winning method.
Scientific research is about truth-seeking, so many researchers are in particular interested in the second interpretation.
I don't disagree.
But what a scientist would do after having strong evidence that huge scaling might help is attempting to understand what part of the much larger complexity leads to this qualitative change.
Remember that Einstein told journal editors to piss off [1] when they tried to get his papers peer reviewed.
[1] https://theconversation.com/hate-the-peer-review-process-ein...
What the people who critique the publication process are missing: 90% of submissions are crap - unfit for publication.
We need some process to gate-keep.
a) The venue of publication is a good signal, whether the time to read a paper is well-spent. b) The PhD students learning the craft need objective feedback. The supervising professior/university often has the incentive to "just submit" -- even if they know that a publication does not meet the quality standard.
Before peer-review, somebody also needed to make a decision on what to publish. This typically fell to a single individual. The editor or some well-known member of the community who could recommend a paper for publication. On old journal issues they even mention the "recommender".
So the question is not whether peer-review is bad, the question is which alternative gate-keeping process would be better. Otherwise we will drown in crap publications (even more) and the PhD students don't get a honest feedback signal upon which they can improve their craft.
> We need some process to gate-keep.
Journals, when print was the medium through which academic research was disseminated, had to gatekeep because there were practical considerations regarding how many articles they could put in each issue. With online repositories like arxiv, this is hardly a concern anymore.
Someone putting a crap article on arxiv does not hurt anyone else, and I'm saying this as a person who recommended tons of articles to be rejected because they had atrocious grammar/spelling issues. Worst case, it gets 0 attention and is ignored by the research community.
Something not being published in a journal/conference proceedings clearly does not prevent it from drawing tons of research attention, as we saw in numerous cases like the Adam optimizer [1].
Which brings us to the second point: what even is the purpose of a journal now? The answer is that the sole function of a journal now is gatekeeping, with the presupposition that, as you observed
> The venue of publication is a good signal, whether the time to read a paper is well-spent
Except, well, top journals have tons of articles that get 0 citations too. Clearly the filter fails at this purpose as well. So, why gatekeep at all then? Because if we did not have some exclusive prestigious journal, the plebs would not be separated from the esteemed titans of academia with the biggest grants, most prestigious scholarships and diplomas from the most famous universities.
The only reason we need to gatekeep today is to feed the academic prestige and politics machine. If you care about the science, upload the goddamn PDF to arxiv , tell your colleagues about your research at a conference and let the scientific community decide on whether your idea is interesting.
That said, there definitely are very relevant papers that are not published in any peer-reviewed venue. A good example is "Language Models are Unsupervised Multitask Learners" (the GPT-2 paper, which I would argue started the whole generative LLM revolution). But I think if you look for this kind of papers, you will find something in common to all of them: they are by very well known researchers, elite institutions or influential companies. That's why people went out of their way to read them even if they were posted somewhere without peer review.
If you removed peer review and just relied on posting to arXiv or similar, new researchers, or researchers from less known institutions, would have no chance at all to make an impact. It's peer review that allows them to be able to submit to a top journal, where the editor and reviewers will read their paper, and they can get a somewhat fair chance.
PS: I don't really like the peer review system that much either. It's just that every alternative that I have seen proposed so far is worse.
Fair, must have misremembered that one.
> If you removed peer review and just relied on posting to arXiv or similar, new researchers, or researchers from less known institutions, would have no chance at all to make an impact.
I disagree on this one. I did my PhD at an institution that ranks in the top 10 in the most well known university rankings, and I distinctly remember that one time when I was submitting a manuscript to a prominent journal in my field, got some reviews back which weren't positive yet were quite valid criticisms, and my professor told me not to worry because the editor is his buddy and my manuscript will get published for sure.
When that sort of "scratch my back and I'll scratch yours" culture exists in journals I don't see how peer review can be an equalizer. It just means everyone who publishes at a journal, including the less well esteemed ones, can claim they went through the rigor of peer review. Of course, we all know peer review is just a vibe check and is actually not that rigorous at all, and besides no one cares unless you published in a prestigious journal anyway. The less revered journals exist to collect $5k in open access fees for the publisher in return for hosting a pdf at the marginal cost of maybe a cent a year.
[0] https://elifesciences.org/inside-elife/66d43597/elife-s-new-...
So the role of journals and conferences is not to prevent "the word getting out". It is to provide value by a curated list of on-topic and high-quality publications.
So no need to wade through tons of crap. Especially for PhD students who might take more time to detect crap as such.
In my experience, the publications at the good venues get a lot more eye-balls and by consequence citations. So there seem to be a lot of people who like this role as a "filter" for what to focus on.
> ... only reason we need to gatekeep today is to feed the academic prestige and politics machine
This to me says you have may not have experienced some parts of the (long-term) research process. It suggests that you have infinite resources to filter out noise, which is probably not the case. It suggests you're willing to spend a lot more time figuring out why something doesn't quite make sense, rather than get to the heart of the problem, while this is fine in many cases it sucks when you're hot on the trail of something interesting, and you're slammed by a million twisty paths full half-baked hot-takes.
We need to filter ("gate-keep" is pretty inflammatory term) information and processes so that we don't have 12 different screw types with 12 different electric screwdrivers, instead of "just" 6 (sigh). We need to come to consensus and that means some things go in and something are left out. We need many mechanisms to filter.
> tell your colleagues about your research at a conference and let the scientific community decide on whether your idea is interesting.
All of these things feel like filters, when does a filter become a gate: colleagues - i.e. not everybody, but some selected few, who and how?; conference - filter (well, gate!); scientific community - != your baker's community; decisions directed by you, not on my own (i.e. a pointer to my paper) - filter.
[Edit for formatting, sort of.]
As for the "scientific community" being a filter, there is a difference between "elite" researchers being the ultimate arbiters of scientific truth via their positions in the editorial teams of journals versus everyone being allowed to publish on open platforms like arxiv and bad ideas/quackery being filtered out naturally.
Because the former is what makes or breaks a scientist's career, grad students and postdocs hyper optimize to publish at prestigious venues, as opposed to optimizing for doing science. These two are aligned only sometimes.
Per Goodhart's law: "When a measure becomes a target, it ceases to be a good measure." [1]
[1] https://archive.org/details/ImprovingRatingsAuditInTheBritis...
Nor do journals.
I think it's straightforward to make an argument that many of today's conferences are as bad as journals, accepting submissions is only one way conferences filter. IRL they are prohibitively expensive to enter, let alone attend (but again, see "Zoom"), and therefor eliminate all but the elite, they are run by commercial entities in all cases with more than ~200 people, they are more or less required venues for networking and therefor selling yourself for a tiny chance at academic permanence, they give plenaries to elites (filtering to one voice), they have special symposia by invite only, with other submissions dumped to inaccessible parallel sessions (which one will you choose to see?), the submissions you make are published in much more ephemeral ways, and tend to be more difficult to discover in the long term, making the event important but the research not as much (at least in my experience), etc.
But having a browsable collection of the verified non-crap articles on any given topic helps most everyone working in that area.
Publication is antiquated. HN doesn't need reviewers to boost the best content or to provide commentary on how to improve a paper or fact check its contents. Join the 21st century.
Sorry, don’t mistake my tone for condescension. I just wanted to explain how this looks to an outside observer.
Academic politics always existed of course. We should not be under any illusions that the greats of the past just wrote their manuscripts in isolation. However it was not an industrialized machine like it is now, and incentives are misaligned in academia to such a degree that the machination of academic politics killed the reason why academia exists in the first place (scientific discovery).
To advance in the academic cursus honorum, one goes to a presitgious undergrad to go to a prestigious grad school, so you can get a prestigious postdoc grant, so you can get a tenured position at a prestigious institution, so you can get a fat and prestigious government grant, which you use to hire bright young students who want to do the same. Note that scientific advancement does not play a role in this cycle, it's actually safer to pursue incremental and irrelevant improvements which you get published through thanks to the connections you made throughout your prestige optimization career.
As a result, academia has produced no notable scientific advances in a long time. It has instead evolved into an organism which selects individuals that excel at funneling money into itself under the guise of doing science while not necessarily doing science (though it incidentally occasionally may).
The kind of behaviour the author is promoting is telling individual prospective grad students you're small, the academic politics machine is so big, yes we all know it's a farce but you just need to suck it up and play the game. In doing so, the prospective grad students will strengthen the machine that is actually killing the very thing they want to cherish and promote, in the hope that they receive a few scraps in return.
(Writing all this as a PhD and former journal reviewer)
He was just as wrong then, as you are now.
1. Finish
2. Finish
3. Finish
> Don’t try to reform science. Not yet. Not in your PhD.
So, I don't agree with that. While yes, you won't succeed - without this kind of idealism, academia is doomed.
It's only when we enter into the domain of unfalsifiable things that we enter more into systems of belief than systems that can challenged or tested. So for instance most of social science is not scientific, because the concepts are generally entirely unfalsifiable. The "Journal of Personality and Social Psychology" is one of the leading journals in its domain - and also one of the most frequently cited by the media due to its oft catchy headlines. It's also one of the best examples of the replication crisis where only about 20% of papers published in that journal are able to be replicated.
Does that mean that 80% of the papers in it are thus fake and false? Nope. Because the entire domain is just completely unfalsifiable, so the complete inability to replicate the overwhelming majority of what that journal claims has done little to change its premier place in social psychology. It's just entertainment with some standard deviations attached.
So looping back to religion, the issue isn't the methodology. It's the lack of falsifiability. You simply cannot disprove the concept of e.g. a spirit because it's inherently immeasurable, untestable, and unknowable. Yet the lack of falsifiability does not mean false. For instance the exact same is true of consciousness. If I claim you're a philosophical zombie [1], you can't prove I'm wrong (or right), because the entire notion of consciousness is unfalsifiable.
Do you mean by artifical means or sensing machines or something?
Because spirits are a metaphysical but very human concept, and I answer that they can be perceived, discerned, described, and known [perhaps not fully or objectively].
Faith and reason are both in operation for science and religion. There is a complex history of falsified doctrines, miracles, apparitions, and communities for mainstream Christianity, as I am sure all others.
You can't dissect a Eucharistic host or throw it into a Mass Spectrometer and find Jesus molecules, [unless you're Carlo Acutis] but if one billion people can tell the difference, who are we to judge?
It's better known than Prozac, and possibly more effective, what do you think?
And no idea if the reference to Prozac was random, but yeah that is a good example. It's effect is not falsifiable at all. It's based on self reporting. And controlled studies, particularly those not carried out by parties with a vested interest in the outcome, shows its effect be scarcely better than placebo. Some might cling onto to that with 'well that does mean it's still better!' yet Prozac has extreme, and rather rapid onset, side effects which makes it basically impossible to not know if you're getting the right thing. This completely ruins double blinds, and could largely explain what little effect it does show.
This was different in the past because back then academia was vastly smaller, so practically everyone from post—doc on up was a literal genius, or close to it. And thus much harder to replace.
Grad students try the hardest to change things because they are the most affected. The problem with "tabling the issue for later" argument is that you just keep doing this and we end up with exactly the system we have. Maybe it isn't a PhD to do, but there's always something. Professors are still overworked.
There was a good post on BlueSky recently[0] that quoted from the instructions for reviewing PNAS
The purpose of peer review is not to demonstrate proficiency in identifying flaws
But I think there's a fundamental alignment problem. An irony in ML, since surely this is far easier than the AI alignment problem. But the purpose of publishing is to communicate. Are we actually doing that? Is our review process improving communication? Or is it actually just gatekeeping or blocking out voices? It is one thing to reject works because they communicate poorly, don't evidence their hypothesis, or are outright fraud, but why are we blocking anything else? This stupid notion of prestige? That's never going to end well.
Not to mention all the wasted time and money...
[0] https://bsky.app/profile/docbecca.bsky.social/post/3lkbec2hi...
> You can't just say "I actually have no idea how to do research in this field
I'm not sure why you jumped to the conclusion that I'm not part of this field.
Survivorship bias.
The people who get to the top are those who can play sufficiently well by the rules of the system.
You can let it happen or resist its influence over you from the start. Maybe through reform. Maybe by doing your own thing.
As another commenter mentioned, survivorship bias. "It sucks, but it is working, right?" Often we want to convince ourselves of this because it can justify the bad stuff that happened. We want to rewrite this in our heads because it helps us not get depressed. But there's always room to improve and I think this is the better aspect to focus on. Sure, what we're doing may work but that is not a reason we shouldn't improve.
In fact, one of the most frustrating aspects to me is that it is the job of a scientist/researcher/engineer to find problems and then improve them. So that's why I find maintaining the status quo rather infuriating. It is in direct opposition to the fundamental framework we work in: to always be improving.
https://osf.io/preprints/metaarxiv/pkahc_v1
^ This is a review I co-authored of a meta-science book that focuses on the scientific enterprise. (The book is oriented more to life sciences than compsci). Although my personal preference of a term is 'praxis of science', I go with one of the more frequently used terms. But I do appreciate the enumeration approach that inherently juxtaposes Science 1 vs. Science 2.
Please post about your experience! I'l love to read what happened
Anyway, I have enough money to fund myself for at least a couple years, so my goal is to make the best of that time and see if I can upend the dominant paradigm in NLP (humans don't need as much data to learn — I'm going to pursue ideas that allow computers to be as data efficient).
Having said that, I now work in industry research and frequently find myself having to remember and remind others (particularly junior researchers) not to fight the system of industry research too hard if they want to actually get anyone to take notice of their research.
It's rare to find people who are both good scientists and good politicians.
I would focus on staying sane psychologically and socially, and not feeling I'm wasting my years as much as possible. Though if you can participate in some movement of your colleagues with potential for a positive change, do that. Just don't overburden yourself alone.
I kind of think academia needs to be reformed and unwound from the outside. But I am more and more convinced by the news that the 'outside' might be unable to do it in ways that aren't appalingly dumb. So my faint hope would be that the much maligned "gen Z work ethic" will eventually force some change, and the old academia barons just die out.
At the moment, yes it is; but when HackerNews buggers off, things will be back to normal.
Since I do this commercially, not for science, and since I don't agree with software patents (on principle), I don't "describe" the method anywhere.
Now it's possible that in that code is some novel thing. Perhaps am insight, perhaps an algorithm or technique. (I don't really think there is, such is my modesty, but its possible.)
Obviously from a "science" point of view, my work is meaningless. And perhaps that speaks to the point of academic science versus the "real world". Their goal is to "write stuff down" - not "do stuff" as much as "record stuff".
In the context of the above quote, I believe we might be doing scientific things (even novel things) but if we're not actively sharing that knowledge it's not "science".
Now of course lots of us -do- document things in blogs etc. But this informal writing on the internet is adding to a haystack, in which there might be gems, but how can they ever be found?
Give Copilot and the others some time to chew through GitHub, I guess. Perhaps one day, an LLM will write the successor to Chrome.
Throwing more parameters at LLMs is no longer yielding appreciable improvements, and once you have thrown all the data at them you can find, that dimension is done also.
Startups work best when the product fits perfectly into the market need. This requires meticulous investigation into the scope and shape of the market need as well as what approaches (business model, product, onboarding, value prop, etc) best satisfy it.
Observe -> theory -> hypotheses -> test -> repeat
Submitted it for discussion at https://news.ycombinator.com/item?id=43403596
An advisor who knows what the student wants to accomplish, and is capable of accomplishing should be able to determine when building a working system is more valuable than pushing yet another paper, when reforming science in a small way is likely succeed, and so on.
Broadly speaking, I agree with the author's point, that one needs to learn the rules of the game before trying to futz with them, which means one will ultimately be more effective learning the ropes in the first few years of a Ph.D. program. Then, after, one will be in a much better position to change things.
One big issue I see is that the skills that academic training engenders is almost orthogonal to management. And, unlike most of human history, we are now in an Information Age, with both private and public knowledge production economies. The private knowledge economy (i.e., tech, broadly writ) utilizes many practices that are barely heard of in Academia. Minor case in point: project management software is not the norm, at least not in my field.
For those who are interested in this topic, there's a very interesting set of proposals for how to bring "Science 2" much closer to "Science 1" in Michael Nielsen's and Kanjun Qiu's monograph / book "A Vision of Meta-Science." [1] Fair Warning: it is very, very long. But the first part is quite short and proposes a number of interesting Science 2 reforms that should interested HN readers. Tenure Insurance (proposed by none other than Patrick Collison), funding by grant-rating variance, etc.
I'm still finishing the essay, but so far it's the best thing on the state of science I've read to date.