Cargo Cult Science (1974) [pdf]
calteches.library.caltech.edu
calteches.library.caltech.edu
Sometimes people now claim that "cargo cult science" is not really a good analogy and should be abandoned.
However, I think this newer understanding of cargo cults may actually make it an even better analogy.
Even if a line of scientific ideas is mostly fake and its research practices can't possibly lead to truth, participating in this ritual of fake research, giving talks about it, and other science-shaped activities, still does bind the participants together. It lends prestige to the leaders of the field. It gives everyone a way to coordinate politically around securing funding and legitimacy from higher powers for their fake research area. And we've seen you really can keep a field going this way for a very long time even if the planes never land.
I've often used "cargo cults" as a way to categorize stupid ritualistic things about the world I didn't like, but it's good to be reminded that the world isn't so two dimensional.
The idea is sound, but don't assume you understand the members enough to do a good job.
>> I think a better example of a cargo cult than bad science is climate change denial. The followers see the potential loss of "cargo" in the form of their current way of life as a problem and the leaders apply the skills that brought them success in politics to that problem. Someone good at reading legal documents is going to think that there is something wrong with scientific papers.
Specifically: "The followers see..." - it is not known what climate change "deniers" believe - rather, people tell each other stories about what they believe, and believe these stories to be true. Some humans are currently smart enough to realize that this behavior is flawed in certain scenarios (race/gender/etc stereotypes), but when it comes to ~political stereotyping the mind (and mainstream "right thinking" culture/media/etc) will defend the delusional practice to the death.
Typically these topic are argued primarily using memes (Gish Gallop, Whataboutism, Brandolini's Law, etc) and rhetoric (rendering it not possible for a strict logical perspective to "win"), with some stats and studies included to give the appearance of objectivity, permanently locking humans into this ~conceptual Overton Window.
If humans could analyze and evaluate the system we are embedded in with the same emotional/psychological detachment we analyze computer systems with it would be a big improvement, but I do not see it happening anytime soon.
More than one capital-A style agile consultants have told me that they don't really believe their scripture but the job pays well.
I can believe that. I work with Kubernetes-shaped objects, and the rituals we go through can not in any way be technically motivated. But there are conferences held and new software written so who cares, really. Nobody really needs this stuff and we all know it.
You just described modern US politics.
As with any and all human institutions, even those started with the best of intentions, it is surely just a matter of time before money and power corrupt and bend whatever-it-is to server money and power.
It seems like we have entered the explanation phase where reasons are discovered where the stupid and pointless thing actually had some minimal benefit that in no way was worth the effort expended.
I think this is generally a good sign as it means the pain from the event is fading and so people can reflect on it in sort of silly and absurd ways.
And it's in many places, if not everywhere. People mock the Polynesians for engaging in a Cargo Cult but they don't realize that most countries do that in more "sophisticated" ways.
Think about the big infrastructure projects that some "dictators" do in some countries. It would make more sense for the dictator to just pocket the money, or distribute it equally if he wants to give it up. But these projects and their prevalence suggests another dynamic at play here.
I mean, ultimately infrastructure is an investment. Now, it might be a _bad_ investment; some infra projects don’t make sense. But there’s a fairly obvious reason to do them, and even in totalitarian countries many are good investments.
I think a better example of a cargo cult than bad science is climate change denial. The followers see the potential loss of "cargo" in the form of their current way of life as a problem and the leaders apply the skills that brought them success in politics to that problem. Someone good at reading legal documents is going to think that there is something wrong with scientific papers.
A climate change denial group in the UK [1] rented office space from one of the scientific institutions that used to meet in Carlton House Terrace. The GWPF held meetings where they could talk in a sciency way even though they were not reporting any real science that they had done themselves.
[1] https://en.wikipedia.org/wiki/The_Global_Warming_Policy_Foun...
Basically the gist as I understood it is that when Christian colonial missionaries arrived on Tanna Island, they banned all the local religious practices and quickly established themselves as the sole administrator and authority on the island (giving out harsh discipline and torture basically to maintain this power).
Once WWII comes around, all the missionaries fled, and soon the Americans arrived bringing with them cargo and gear and importantly, employment and autonomy. Locals were used as couriers and carried supplies in support of the Americans, and in return were given "goods" that were to them simply unheard of before.
When the Americans left, the colonial missionaries returned, but because so many people gained this autonomy and ability to practice their former religious ideas, this "cargo cult" formed which essentially was formed by anyone who refused to partake in the Christian authoritarianism that was running the island. People joined simply because it allowed them to have freedom and autonomy and the movement basically fused previous religious practices with these rituals that the Americans brought.
The "dream" for John Frum (i.e. the Americans) to return really meant a dream for independence and the Christian missionaries to leave.
That said, I thoroughly agree with your last paragraph.
While there are legitimate concerns about the scientific method being too strict, I think it's necessary to help us keep faith in a field. That's why I dislike fields that lean into methodological anarchy, like critical theory. They really have no business claiming any sort of authority.
> The amount of energy needed to refute bullshit is an order of magnitude bigger than that needed to produce it.
The logical conclusion of that is the Gish Gallop:
https://en.wikipedia.org/wiki/Gish_gallop
> The Gish gallop /ˈɡɪʃ ˈɡæləp/ is a rhetorical technique in which a person in a debate attempts to overwhelm their opponent by providing an excessive number of arguments with no regard for the accuracy or strength of those arguments. Gish galloping prioritizes the quantity of the galloper's arguments at the expense of their quality. The term was coined in 1994 by anthropologist Eugenie Scott, who named it after American creationist Duane Gish and argued that Gish used the technique frequently when challenging the scientific fact of evolution.[1][2]
Even if it's a moderated debate where an outright gallop isn't allowed, it's still easier for the dishonest party to erect a strawman claim than it is for the honest one to carefully rebut it, especially if the dishonest claim is so far into wackyland it's Not Even Wrong and requires a lot of its assumed premises to be demolished first.
> While there are legitimate concerns about the scientific method being too strict, I think it's necessary to help us keep faith in a field.
Important phrase: "keep faith in" - faith is always in play (it is fundamental to culture & consciousness), and it is healthy for the institution of science to acknowledge it imho.
> That's why I dislike fields that lean into methodological anarchy, like critical theory. They really have no business claiming any sort of authority.
Might this be an instance of "You only need to lie about or misrepresent a few facts to undermine a whole field, and to an outside observer it looks like {the field} got conclusively btfo"?
You are discussing your opinion/interpretation of critical theory, necessarily, but it would be very easy for readers (or even yourself) to form a belief (which is typically perceived as knowledge) that your subjective evaluation is objective and accurate.
Science and its methodologies are excellent for figuring out the physical realm, but watch out if you apply these methods to the metaphysical realm.
Science does rest on unfalsifiable foundations, but this isn't productive to discuss. I'm talking about the faith that scientists aren't making stuff up given the reasonable foundations we all already agree on.
> Might this be an instance of "You only need to lie about or misrepresent a few facts to undermine a whole field, and to an outside observer it looks like {the field} got conclusively btfo"?
I don't think so. If you accept my premise that authoritative fields should be held to strict and conservative scientific standards, then critical theory is categorically ruled out by it's own definition. Yet people dress it up as if it were strict and conservative field, like a cargo cult.
> You are discussing your opinion/interpretation of critical theory, necessarily, but it would be very easy for readers (or even yourself) to form a belief (which is typically perceived as knowledge) that your subjective evaluation is objective and accurate.
I think I made my premises clear, but I dislike the suggestion that they are merely my opinions. Most people actually share my views, but are willing to pick and choose when they apply them.
> Science and its methodologies are excellent for figuring out the physical realm, but watch out if you apply these methods to the metaphysical realm.
I'm not applying science to critical theory. My issue with critical theory is it's methods.
How could you possibly know such a thing? What does "not productive" even mean in this context?
> I'm talking about the faith that scientists aren't making stuff up given the reasonable foundations we all already agree on.
Ya, me too - is it not weird that those who subscribe to science because it "does not" deal in faith do so based on faith?
> I don't think so. If you accept my premise that authoritative fields should be held to strict and conservative scientific standards, then critical theory is categorically ruled out by it's own definition. Yet people dress it up as if it were strict and conservative field, like a cargo cult.
"Thou shalt have no other gods before Me"
See also: Meme Magic.
>> You are discussing your opinion/interpretation of critical theory, necessarily, but it would be very easy for readers (or even yourself) to form a belief (which is typically perceived as knowledge) that your subjective evaluation is objective and accurate.
> I think I made my premises clear, but I dislike the suggestion that they are merely my opinions.
Where did this word "merely" come from?
> Most people actually share my views, but are willing to pick and choose when they apply them.
You have no means of knowing what all people believe.
> I'm not applying science to critical theory.
What about: "If you accept my premise that authoritative fields should be held to strict and conservative scientific standards"?
> My issue with critical theory is it's methods.
Technically: your impression/model of its methods.
[0] https://en.wikipedia.org/wiki/Bill_Nye%E2%80%93Ken_Ham_debat...
A few examples that come to mind:
* because lots of famous VCs made contrarian bets, new VCs try to be contrarian (even when a consensus point of view is clearly correct).
* almost every startup I know is looking for 10x engineers, even though most startups don't need 10x engineers. We've just all been conditioned to believe that 10x engineers are required to build a great company.
* generalizing the 10x example, young startups copy the traits of FAANG companies or famously successful startups, even if those traits are harmful for early stage companies.
Feynman: "The first principle is that you must not fool yourself—and you are the easiest person to fool."
Also, here's a text version of the OP that's easier to read on mobile: https://calteches.library.caltech.edu/51/2/CargoCult.htm
People think that if they keep saying nice things about Apple, they will not one day lock them down hard.
The iPhone was hot when it came out, and was a lot of people's first taste of the smartphone, and it's very easy to fall into tribalism. You and I aren't immune to it, there are hundreds of things we are irrational about in our lives. Arguments in favor of your chosen tribe are proxy arguments for preserving your own value, identity, and dignity.
People who rebelliously use non-Apple products are helping to keep the alternatives alive. It's a tradeoff of now vs future. That's the real tradeoff, it's not cults.
(edits: HN vs who?)
There was a time when companies like Google were looking for very talented CS people because they actually needed people with broad skills because in the case of G they were building a search engine and there's almost no area of computer science that isn't involved in such a project. They actually needed people with strong CS skills.
Twenty years later we have positions where hires are selected for their ability to reverse a red-black tree on a whiteboard, where the position will mostly be about gluing together CRUD apps with YAML.
A few years ago I worked as an interviewer for a large software engineering recruiting company. We did quantitative scoring on a lot of parts of our standardized interview. We had sections in our interview on CS knowledge, programming, debugging and whiteboard style problems. Based on the data, we asked: Could we eliminate any part of the assessment? Could we throw out the CS knowledge part without losing accuracy about the overall hireability of the candidate?
Based on the data, the answer was no. The scores were all positively correlated - so CS knowledge implied you were better at programming and vice versa. But we still got extra signal by assessing candidates on their CS knowledge. Turns out even if you aren't amazing at programming, having excellent CS knowledge will still make you desirable to a lot of companies.
(The weakness of this study is we didn't follow up with people. We only knew if our candidates got hired, not how well they did after they were in the door, as employees. So we might have just been mirroring the same biases the companies themselves have in their hiring processes.)
This is some weakness. Surely this is the outcome that's interesting.
I also suspect there's likely a correlation between how hard someone is trying to get hired and the freshness of their CS skills.
Chances are a motivated candidate will have been brushing up on CS because it's such a trope to be grilled on those types of questions; that same candidate likely prepared for other interview questions as well and I would indeed expect that to increase the odds of getting hired.
Since nobody walks around with perfect recall of the types of algorithms that crop up in the classic tech interview, it's fairly safe to assume the CS part of the interview is a direct measure of how much preparation the interviewee has done.
I'd love to have data on that.
I know its really common for engineers to never touch this stuff in their day to day work. Most product teams don't need any of this knowledge at all. So asking about it in an interview is a massive waste of time.
But personally, I've used a lot of these algorithms while working on collaborative editing for the last few years. For diamond-types, I ended up writing my own b-tree and skip list implementations, and I make heavy use of binary search, BFS, DFS and priority queues. I've used priority queues in plenty of projects - like, years ago I made a library to mock out timers in nodejs for our test suite so we didn't have to wait for real timeouts to trigger in our test suite.
But I've got no idea what percentage of working engineers use any of this. 5%? 1%? 0.01%? On the surface, it seems nowhere near useful enough to justify how often these questions show up in interviews.
Just because I've implemented a binary search a few times, and a b-tree, and a skip list, and various sorting and intersection algorithms doesn't mean I can reconstruct them on a whiteboard from memory. What it amounts to is that I have an upper quartile understanding of the underlying idea and the general quirks (among practicing programmers), but not much more than that.
If you would fail an interview like this, it’s not calibrated correctly. I know it’s unsatisfying, but nobody gets perfect marks on assessments like this by design. Nobody is expecting you to program up a correct btree on a whiteboard in an interview. Just being able to speak in detail about data structures and algorithms from practical experience is a very strong signal in a candidate. Let alone being able to explain high level concepts, and talk about when they’re useful and maybe explain some implementation details. That’s great!
In my experience interview processes act like a series of stage gates, you have to pass each and every one to get hired. It seems trivial to say the doing well in each activity is predictive of success when you need to do well in everything to be successful.
I don't mean to be mean, but that study sounds like a giant missed opportunity to do something useful. Knowing what interview activities are actually predictive of being good in a job is the holy grail.
Just lol
I’m sorry, but then your study doesn’t show anything useful. Interviews are supposed to determine whether someone is going to be a good employee, not whether they are “desirable to a lot of companies.” So yeah, all you were doing was mirroring the bias of those companies’ processes. As a recruiting company, that can be good business, because you’re giving the customers what they want. But it’s not effective hiring practice for the customer.
Furthermore, you did not follow up on the people that were weeded out to see if they would have been good employees. Current hiring practices weed out a lot of people that could have provided a lot of value, but they weren’t able to perform the prescribed ritual on cue, you didn’t have a chance to evaluate whether that was a good decision or not.
Kudos for the self-awareness but like others have pointed out, cmon.
This obsession with quantitative scoring for tests is a great example of a cargo cult science because you are doing all of these intricate little rituals that amount to nothing because you're simply measuring latent social phenomenon of hiring like-minded people. It's just statistically-laundered-bias.
That's not just a weakness, that invalidates the whole study.
Their point isn't to test a specific practical skill.
"20th Century psychiatrists wrote manuals and procedures for the diagnosis of ‘mental disorders’. They order lab tests and undertake careful study of their patients’ symptoms, and they diagnose the conditions listed in their manuals. They seek regulatory approval of prescriptions to treat the diagnoses. They stopped doing psycho-surgery by mid-century, but still electrocute their patients’ brains when they think the patient will benefit.
They have their own special hospitals, where troubled patients are sent to be stabilized on palliative prescription drugs. The psych wards have little areas for staff to sit in. The staff wear scrubs and badges, keep medical records, and some prescribe prescriptions—[s]he’s the doctor—and they wait for their patients to get better.
They’re doing everything right. The form is perfect. It looks exactly the way it looks in the other medical specialties. But it doesn’t work. No patient recovers as a result of the allopathic drugs provided. Some patients get better anyways—perhaps because they’re fed, perhaps because they’re kept sober—compounding the profession’s confusion."
It's true. People get better in psychiatric incarceration, because there is a predictable, regular daily schedule. There are meals on schedule. There are groups and stuff. You see the same people all the time. You're off TV and you're off the computer, so you need to face reality. You're relieved of burdens such as bill paying, cooking, cleaning, and running your household. You might color by the numbers or work a puzzle, but you've got reality, and it's stripped down and reduced to its essential components. You tend to get better and saner in this environment, if that's your baseline and that's where you tend to go. If your baseline is insanity, then all this regularity may not help.
The reason the British Houses of Parliament feature so many exterior and interior chimneys was to funnel so-called 'bad air' of the Thames away from the people who work inside the building. This was based on the newly acquired knowledge that some diseases are passed on via microbes. They wrongly assumed that most malicious microbes were airborne and that bad smells were an indication of their presence.
The history of science is littered with such examples of 'good science/wrong conclusion'. We would be foolish to assume that this is not as true today as 100 years ago.
https://nonprofitupdate.info/2010/10/21/10-leading-causes-of...
But the idea of 'bad air' (miasmic theory)' though Informed by microbe theory, had its roots in Hippocratic thinking which was derived from the observation that swampy areas were generally unhealthy to live. This was attributed to their bad smell and not the preponderance of mosquitoes (malaria).
Fyi... The same miasmic theory gave rise to toilets that were designed to flush away bad smalls with water.
"Malaria was prevalent in the Roman Empire, and the Roman scholars associated the disease with the marshy or swampy lands where the disease was particularly rampant.[13][14] It was from those Romans the name "malaria" originated. They called it malaria (literally meaning "bad air") as they believed that the disease was a kind of miasma that was spread in the air, as originally conceived by Ancient Greeks. Since then, it was a medical consensus for centuries that malaria was spread due to miasma, the bad air." [1]
I found Robert Whitaker's books invaluable for helping me understanding the mental health industry's mindset. Mad in America covered how the Quakers got much better results with their Asylums (safe place with 4 meals a day & activities) than modern techniques of lobotomy, ECT'ers and palliative prescriptions. Anatomy of an Epidemic makes the case that palliative psychiatric prescriptions take an episodic condition and make it chronic. Psychiatry Under the Influence chronicles how psychiatry got captured by the prescription drug cartel. Highly recommended.
The good news from the mental health world is that Chris Palmer, M.D. published Brain Energy [0] last year. I haven't read more than excerpts from google books [1]. My understanding is Dr. Palmer was a conventional palliative psychiatrist, then he had a patient whose schizophrenia improved on a ketogenic diet. The patient was able to discontinue antipsychotics: Dr. Palmer's mind was blown. Then he discovered the 70+ years of research establishing that mental health conditions are metabolic problems, and wrote a book.
Perhaps the most bold and disruptive aspect of
Brain Energy is understanding precisely how and
why medications that harm metabolism might
reduce mental health symptoms.
The long-term consequences are of great concern
and require the urgent attention of the psychiatric
community.
Dr. Palmer recommends a ketogenic diet. I think most of the benefit of 'keto' stems from avoiding fortified flour: https://twitter.com/JamesKnochel/status/1595562197412851712
A lot of people are harmed by folic acid, and the iron and lack of fiber in fortified white flour is harmful for almost everyone, in that they disrupt the microbiome. Most people get adequate iron, and don't actually need iron filings ("reduced iron"); the fiber in whole wheat fiber feeds bacteria that make short-chain saturated fatty acids [2].
[1] https://books.google.com/books?id=FoxlEAAAQBAJ&pg=PT233&dq=B...
[2] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4756104/
(edited to add [2] and some speculations on the mechanisms of harm of fortified white flour)
Can you expand on that? I'm not involved in the field, but there certainly exist many low qbit machines with varying technologies around the world that can already be used for calculations. Yes, they're nowhere near big enough to be useful yet, but qbit counts in the biggest machines are multiplying every year similar to Moore's law, so that seems to be just a matter of time.
That's pretty much my take on it too, just because it's pretty limited right now there appears to be healthy growth in qbit counts which would suggest a more usable quantum computer is mostly a matter of time and further research.
It's not entirely accurate to say nobody knows how to do this. The theory of quantum error-correction has been rigorously developed since the 1990s, and experimental implementations have shown that improved fidelity is possible with known error-correction schemes. I think we'll know definitively within the next 3-5 years how hard it will be to engineer fault tolerance, and workable solutions will emerge from those findings.
After almost thirty years of development, ion traps have now reached a few ten qubits with reasonable fidelities. However, this requires shuttling the ions between trapping zones which is enormously slow (many milliseconds per gate). And scaling this to hundreds of thousands of qubits is a completely open question (none of the current techniques will work).
I'll believe it when I see it. Yes, machines exist, but so far I have not seen one single convincing application that makes it better than a classical computer. Not saying it can't be done, but for all the hype around quantum computers, so far the results are dismal. (And I used to work in a relevant field.)
2001: Shor's algorithm was demonstrated by a group at IBM, who factored 15 into 3 × 5
2012: factorization of 21 was achieved
2019: an attempt was made to factor the number 35, but the algorithm failed because of accumulating errors
I think this is partially a symptom of cargo culting the SV model, like you suggested, along with the much more fundamental reason that software has a very low barrier to entry. Hardware is hard, but it's really the most impactful investment to make in quantum tech at this early stage because of the spin-off applications in sensing, timing and networking.
Look at Uber for instance. Aside from one blip in 2018, it has never had a single year of profitability. Yet it receives year after year of VC funding and investments because surely any day now they have to turn a profit, right? All these people couldn't be wrong...right? I mean this is the revolutionary business that was a mArKeT DiSruPtOr and deregulated entire vast swaths of industries to do it, so surely their success will materialize "any day now" into profit. Afterall, its the most free-market principles-based business right?? It overthrew the tyrannical state-regulated taxi industry! It HAS to be more efficient and better! Except it isn't.
Quantum computing isn't a "cargo cult" of VC startup business models, its simply another VC startup business models like all others hunting for sustainability through venture capital funding.
Now, the magnitude of the error would depend on how long the cargo cultists kept doing this, even though the planes didn't return, and I have never heard how long that was. If they tried this for a month or two, it's not that bad an error, really; it was worth a try, based on what they knew. If they did this for decades, that would be a serious error.
Assuming they didn't get other benefits from the rituals, namely the same ones people get from more mainstream religions like social cohesion and a sense of purpose and a belief in some greater power.
It was working though. It was maintaining a new state of social order, which is a mandate for sustainable power transfer in society.
Feynman's speech is interesting and all but like so many similar gifted scientists who then go on to explore into other domains, they stumble into a field and ignore whole bodies of research into the philosophy and sociology of how science is produced.
Thomas Kuhn had already established some solid insight into the social and organizational aspects of "scientific revolution", and showed that despite most people's ardent belief that science is somehow immune to it, there is a significant amount of social hierarchies and ritualistic beliefs that dictate what is considered "acceptable" science.
Believing in science as this perfect immutable meritocracy is pretty much its own ritualistic cult, as if you can just pretend away human bias and insular beliefs by thinking really-really hard.
Feels appropriate to the current LK99 discussions
It's too bad that he so diminished philosophy of science, and at the same time put so much undeveloped thought and prose into it.
> way too smug sometimes
Immediately after:
> This speech is essentially a philosophy of science piece, at the intellectual stage of at least one hundred years prior, and probably more like three hundred.
Bruh.
Apart from the hypocrisy there, the fact is Feynman did science. He did more science than Popper, Kuhn, and Hume put together. He understood it on a level deeper than >99% of other scientists, and >99.9% of philosophers.
That he did so with a "three hundred year" out of date view doesn't really reflect well on PoS's utility for actual scientists.
Let people who are that capable and accomplished have a blind spot once in a while. What is this trend of cutting legend's ankles gonna accomplish for anybody.
Its a sort of elitism and inferiority complex.
The fact that Feynman was working through some "naive" positivist worldview and yet achieved such success just rubs it in more that a talented scientist needs philosophers about as much as a bird needs ornithologists to know how to build its nest.
When talent, curiosity and integrity come together in this way, it doesn't need some philosophers musings and rulebooks to do great.
How's that global warming thing coming along?
Let me guess: it isn't relevant?
Feynman may not have cared about philosophy, but plenty of other scientists did and do. Plus, philosophy doesn't necessarily have to justify itself in terms of "utility", because that just assumes that everything needs to have some ulterior motive and can't just be enjoyed for its own sake.
Feynman was lucky enough to be a physicist in an era when there was much new, experimentally testable physics. Experimentalists discovered new phenomena. Theorists could propose theories, which were then confirmed or rejected quickly. Most results were clear, not near the noise threshold. The field progressed rapidly. Physics was finding, and had found, a set of concise rules that the universe consistently obeyed. Plus, they won the war. Physicists of that era could afford to be smug.
Today, physicists are still banging their head against the wall on dark matter and string theory. Both ideas are not directly testable. Trying to find the foundations is not going well.
Personally, I think we're all winging it almost every moment of our lives. Relative to domain experts, I'm bad at almost everything I do. (Cooking, driving, talking, writing, planning, etc). I'm only really good at maybe 2 or 3 things.
I think we're all like this. And its fundamentally ok. Its how human brains are wired.
Sometimes I imagine dividing all my thoughts between things I've personally invented and things I've heard from others. Whats the ratio? I think at least 95% of my thoughts come from other people. Maybe 99%. Maybe more.
Luckily you can build on the ideas of others without being a cargo cultist. Simply verify the sturdiness of the foundation before you go adding an extra wing to the house.
You use the models handed down by the researchers because they work. They help you make machines and navigate reality and such. There's nothing shameful in that. You don't have the time for research. Time is money after all.
You are not a scientist. You are not interested in Truth. You just want to get from A to B.
Of course sometimes you have to use tools handed to you by others and sometimes you won't understand how they work (I don't know how CPUs work but it hasn't stopped me from programming).
This speech is largely about how you shouldn't willfully delude yourself. I think we've all had that situation where we're hunting down a bug and rearranging the code just so seems to resolve it, but we don't understand why. At a certain point the temptation is high to shrug your shoulders and move on even without fully understanding the mechanics of the fix. But if you do that, it will likely come back to bite you/someone one day. In the case of science it's even worse than engineering because the entire point of the endeavor should be to advance understanding, rather than to get certain results.
Now, "Y after X thus Y caused by X" (see: post hoc ergo propter hoc) is a perfectly fine hypothesis worth testing, but to keep doing that even if it doesn't work? That's the "cult" part.
What Feynman criticized were certain fields of study, which in his opinion claimed to apply the scientific method to get scientific results (and importantly the associated prestige of Doing Science), but which in his view only practiced a facsimile of the scientific method and got merely facsimiles of scientific results!
Now, cargo cult engineering definitely also exists. If you do X – even without entirely understanding why – and reliably get the wanted result Y, that's engineering. If you don't get exactly Y but are able to adjust X to compensate, that's also engineering. But if you keep doing X without understanding why and keep not getting the wanted result Y (even though you might get some different result Z and misinterpret it as Y), that's cargo cult engineering!
Even if someone questions what everyone else believes is true because "scientists said it", they will only be downvoted and censored.
I'm sorry, I can't take anyone that thinks CNET and Wired is a bastion of digital journalism seriously.
It's well known that these sites focus on 80% profit and 20% Journalism. It's been that way from the beginning.
There is a ton of accumulated "dark knowledge" locked up in research groups, all the tricks (like putting the rat corridor on sand, in that example) but they are not really publishable. To publish, you need a clear story, literally that's the word we use when drafting papers. What's your story? A bunch of boring tricks rarely make a good story. And when people do publish such things, they have to also insert some other conceptual "novelty" contribution (typically a non intuitive tweak of the model architecture) to the paper that makes a +0.5% improvement on some benchmark, just to be able to talk about the real practical but ugly things that really make the whole thing work.
Not to mention the replications that Feynman mentions, ie that before you test your tweaked new model, you have to also perform the baseline yourself, you can't just take it as given in another person's paper. This is rarely done in ML, and not just for resource scarcity reasons. Another reason is that it's damn hard. ML systems are very very complex and essentially impossible to exactly reproduce from a paper description. It would even be hard for the same team to do it, if we deleted their codebase and checkpoints and asked them to redo it.
"But open source!" you say. Sure, except that large systems often evolve and the released code is often a refactored version of a haphazard ducktaped spaghetti monster codebase that was actually used, where they manually edited code files between runs, or hard-coded things, discovered some Bug midway and fixed it and compensated for it best they could etc.
These projects must be done in a few months so youre ready before the next conference cycle happens where someone does something related and now you need to rework your story and contribution claim, or at least you now also have to compare and compete with them.
But let's say you took your time and now redid the experiment of the other prior work, but it doesn't agree with your numbers totally. You can email them or open a github issue. They answer perhaps in a week, and say they don't know exactly the reason, or that it was a different code version they actually used but they can't release that one as it's not approved by corporate (from real experience). Of course with your questions you are also terrifying them and your queries may feel to them like threats of pending potential reputation destruction. So they will be very defensive, which will make you suspicious. But they are just some other grad student like you and probably didn't mean any ill.
I've seen a PhD student on Twitter asking what he should do after discovering an anomaly in an Arxiv paper before the camera ready deadline (the finalized version of an article). Should he publicly "out" them if they fail to incorporate his findings. Of course it is absurd and the camera ready deadline cannot introduce significant new contributions or new discoveries, as that would require new peer review. But he was convinced that he's a noble defender of scientific integrity while doing this. I'm just mentioning this because some junior people may read these Feynman pieces and think they should go on a crusade based on often quite scarce information.
But again, think of the sheer volume of works coming out every week. It's unmanageable. If you stopped to interact with every single prior work of your comparison table in such detail you'd take years to write one paper. A PhD student usually has to publish 3 top papers in about 4 years or so. And some will definitely get rejected. The reality is that high-end labs have become paper factories. They have a process, just like pop songs are formulaic. They get a smart person as an intern for example and pump out a paper in 5 months. Exactly how much of this "you are the easiest person to fool" deep self-reflection fits into such a thing?
And yet.
And yet the cumulative effect is undeniable progress. The thousands of low quality papers are simply ignored. They contribute to someone getting their PhD, and that's their true function. But then there is a small set of works that really are excellent. It's just that the publicly available papertrail in the literature isn't really necessarily what has driven it. The papers are more like a shadow projection of the real world work behind the scenes, filtered to please novelty-hungry impatient reviewers and paper-count-rewarding committees. But of course the sheer hardware growth is a big part of the overall success, but the hardware design was informed by the research, and without the model improvements, you couldn't just hardware-scale the state of the art of 1995 to modern computes and expect strong results.
So for sure the spirit that Feynman espouses here still lives on, but it's alive despite all the incentives, and most of what appears as academic science is not really about contributing some truly usable and convincing knowledge, but a demonstration of the job skills of people towards various personal evaluations, like granting a degree, hiring or promotion.
Most people who start with this starry eyed idealism quickly get it stamped out by the system. The important thing is to yield a productive synthesis instead of a resignatory pessimism. Do your best given the circumstances, but also read the room and don't run with your head into the wall.
The fabulous thing is though, that things adapt. The less these hurried processes live up to the ideal, the more the reputation of the label "science" gets eroded. Many people already react with an eye roll when they hear what "experts" and "The Science" have to say. The trust is finite and can run out.
A few major discoveries in physics and medicine led to a giant reserve of public trust over the last century, but it isn't infinite. Immediately after the moon landing, in the space age, science and scifi captivated the minds of everyone and that was probably the peak of it, including figures like Sagan (or indeed Feynman). Then computing brought a new wave of tech but it, and even AI is less of a natural science, and many popular claims turn out to be overblown.
---
Anyway, we have no idea what exactly made the 100 years between, say, 1870 and 1970 so scientifically productive. Because that's the period that lends the weight to the label "science" in the public and hence for politicians. It certainly wasn't the current academic system of journals and conferences and 8-page papers and rushed peer review, h-indexes and byzantine grant application forms.
And whenever something has prestige, people flock to it and want to also bask in it. And it gets inevitably diluted. But the prestige will move on and there will be some other thing next. Something we will call something else than "science".
Up to "undeniable progress" I was with you, but you basically spent the preceding half ish comment describing how the scholarship is shoddy, even unreliable and certainly unreproducible. So how do we know that "progress" has been made? Says - who? If I claim that no progress has been made how can my claim be refuted, other than by pointing to some sort of consensual acceptance of "undeniable progress", and that mainly on twitter?
As to the hardware - it's mainly data and hardware. Algorithmic innovations have been made but those were made now thanks to the availability of data and hardware. It's difficult to propose a new system when there's no computer that can run it in practice.
A big component of it all is certainly what Rich Sutton called the "bitter lesson", i.e. that throwing ever bigger compute and data at scalable models has been the major catalyst of the field.
But I'll offer up another one. There's some kind of shadow scholarship lurking in the background that doesn't have much of a footprint in the literature necessarily. This is of people trying each other's code, poking at stuff and gaining intuition and converging on practices that actually work, regardless of what the papers say. As an academic you develop a sixth sense for what numbers to ignore and what ideas to try perhaps. This is all vague and adhoc and not the clean story of what The Scientific Method is as described in the serious textbooks. But it does lead to the progress we see. It's just not elegant enough to put into papers. So the main story of the paper is often around a minor aspect of the work, because you can't easily make a story around "I sat down and really combed through everything and made sure not to have bugs and paid attention to all the small details others are careless about." But the net result is that the community will go forward with your codebase and your well-working setup. Even if they think that the part that makes it really good is that weird trick that the paper is selling, when that only contributed like 1%.
And it's not like those long tail papers are totally bogus. Their methods surely work and would beat the state-of-the-art from ~2 years ago even under the highest standards of scrutiny. But it's doubtful it's really better than all the other ones that came out around the same time or a few months before.
Overall, despite the shoddy practices required to publish, which introduces a lot of noise into the system, the trendline goes up, because there is genuine effort put into the work. Maybe all we are doing is some kind of unsexy distributed hyperparameter tuning and architecture search. Hinton humorously called it (at the single-person level) graduate student descent (i.e. that the phd student fiddles with the settings until it works well). And we may be doing this on a distributed level. But still, it does not seem to be overfitting, there have been studies on this that appear high quality.
So well, maybe it's not a "science", but whatever else it is, it is working and does deliver results, so nobody feels a need to mess with a running system. Again, I don't insist on the label of science. Call it engineering, fiddling, or whatever, but the planes do land.
But the major catalyst to what? Neural nets today are just as capable as they were 40 years ago, only now we have bigger computers and more data and we can put more lipstick on the pig.
I appreciate what you say, that systems today give better outputs than in earlier years. But you should also appreciate that those are essentially the same systems, with little tricks and hacks that have no clear contribution to performance, since as you say they are unreproducible. And each and every one of them inevitably, inexorably falls to bigger systems with more data and more compute - isn't that the bitter lesson after all? That you can try to be smart but the guy with the bigger machine and the bigger dataset will always win?
And all the while the major limitation of neural nets trained with gradient optimisation remains unaddressed: that they can't function unless they have large amounts of data to train with and they only get performant enough to upload a paper to arxiv, let alone to use in a real problem, once they've been trained with hundreds of thousands, if not millions, of examples, at gigantic monetary cost. That is because -contrary to absurd claims by some- neural nets can't generalise their way out of a paper bag. If they could, they wouldn't need so many examples; they would learn, and generalise, robustly, from few examples. But they can't. Not at all.
The field has simply made a virtue out of necessity, and hence comes the embrace of the magickal shibboleth of the Bitter Lesson. Neural nets can't generalise? Just overfit to more data.
But, imagine if computer scientists, for example, claimed progress because we can now sort ever larger lists using bubblesort on giant compute clusters; or that the question of whether P = NP can now be laid to rest because we can solve simple instances of the Travelling Salesman Problem on a sufficiently powerful supercomputer. That is the Bitter Lesson applied to an imaginary computer science field, where progress had stalled, no sorting algorithms with good asymptotic complexity, like quicksort or mergesort, were known, and where combinatorial optimisation heuristics like Clause Driven Clause Learning where not invented. That is the machine learning field today, having learned the Bitter Lesson: a field that has given up on solving its research problems and instead turned to profit-making schemes to get more money from giant corporations who are in the business of beating people up with millions to make billions, and not in the business of answering any useful research questions.
And you know what? Because of this, progress, even by the self-selected metric of "beats benchmarks", eventually stalls. Machine vision hasn't progressed much since 2012. Machine translation is still stuck in 2016 ish. Reinforcement Learning is coming next year, together with self-driving cars. Now we got LLMs and image generators. Give it a few years and that will be so 2020's and a new peak of hype will be reached, while LLMs remain incapable of telling truth from bullshit.
Science is not an ideal, it's the way to make things work. Like, really work, in the long run. What you describe is the way to only make things sort of work and only in the near term.
DeepL today translates to and from Hungarian much much better than the best systems 7 years ago. You can say it's pure scale, but whatever. Maybe that will be enough. And self supervision is working well nowadays through masked autoencoders and others, for pretraining. CLIP works great. Self driving exists and operates in certain locations. Yes, it's not everywhere yet. Image generation and chatbots are at a level unimaginable just a few years ago.
The bitter lesson is called bitter for a reason. It leads elegance-seekers to despair and they downplay the results. They rather shift goalposts and deny that the thing that has landed really is a plane.
Also, someone has to do the work of designing things like Mask-RCNN or YOLO, their architecture is nontrivial. Anchor boxes, proposal networks, they had to be proposed. Batch normalization, Adam optimizer, learning rate schedules, initialization techniques like Glorot and He, ReLU replacing tanh, and a while host of other tricks. These were significant. Or take NeRF.
If you are expecting such a discontinuous break as Newtonian mechanics, Darwinian evolution, relativity or quantum mechanics, you will be disappointed. Such moments are a handful examples over the entirety of human history. You can take it to an extreme when one scoffs at neural nets that it's nothing new. I mean Schmidhuber considers Gauss fitting a straight line by least squares in the 19th century to be an example of (single layer) neural net machine learning. Technically, sure.
Maybe AI will be solved without any scientifically satisfying insight, but again, that has little consequence to the social transformation it will cause.
You're the one who brought aesthetics into this, not me. I just want to know how things work, and why. That's what science means, to me.
And the process we both, I think, understand is happening in machine learning research doesn't help anyone understand anything. One team publishes a paper claiming this one neat architectural trick beats such-and-such systems on such-and-such benchmarks. Three weeks later another team beats the former team on the same benchmark with another neat trick.
All we learn is that some people came up with some neat tricks to beat benchmarks, and that you can't reuse the same trick twice. So what's the point of the trick, then, if you have to do it all from scratch, and you never know how to make something work?
Neural nets work only when they work. When they don't work, nobody will tell you. Teams who beat the SOTA will gloat about it and keep schtum about all their many failures along the way. Their one neat trick can't beat _this_ benchmark? Nobody will ever know.
The sum of knowledge generated by this process is exactly 0. That is why I say that progress is not being made. Not because it looks ugly. I don't care about elegant or ugly. I don't know why you assumed I do. I care about knowledge, that's my schtick. When there's a few dozen thousand people putting papers on arxiv that tell us nothing at all, that's a process that generates not knowledge, but noise.
>> DeepL today translates to and from Hungarian much much better than the best systems 7 years ago.
Based on what? BLEU scores?
>> The bitter lesson is called bitter for a reason. It leads elegance-seekers to despair and they downplay the results. They rather shift goalposts and deny that the thing that has landed really is a plane.
That's a rotten way to react to criticism: assume an emotional state in the critic and dismiss them as being dishonest. If you can easily dismiss the criticism by pointing out its flaws, do it, but if all you have to say is "nyah nyah you're butthurt" then you haven't dismissed anything.
If you don't trust BLEU and don't trust my gut, how about a randomized survey of professional translators, or how many seconds per character a translator would need to fix up the raw output into something they'd be comfortable with calling accurate? Like, I'm not stuck on a metric.
You see, when I and many other academicscomplain about academic culture, rewies and papers, what we complain about is that on the margins it's kind of nebulous and not at all clear cut whether the proposed method in those thousands of papers are each rigorously tested to a really convincing degree that they really beat the previous approach in some deeper sense, eg if the same amount of hyperparam search went to it, if a broader range of benchmarks were applied etc. But the presence of this noise does not crowd out the signal. Because there is a world outside the formalized systematic, officially presented Science with journals and conferences and awards and h indexes. There is a world of more immediate reality. The mechanized process of awards and publications and grants is not the primary object. I can tell if I'm hungry or full without looking at some figures of hormone tests from my blood.
Yes, we have no guaranteed way to tell in advance what exact architecture and hyoerparameters will work best. Maybe the answer will be very context dependent. But this doesn't mean we are completely lost. The vocabulary of bias and variance, of overfitting, regularization etc do help. There are people who inspect loss landscapes for continuous reachability between different modes, there are hypotheses popping up regarding the sharpness of minima playing a role etc.
Again I don't know how much you are inside the research community and how broadly you are aware of wide range of approaches that exists out there. The goal of "understanding why", and explainable AI are hot topics and there are good groups that come up with results, but none of it has a snappy one sentence summary where you slap your forehead and go "oh, so that's why it works".
So yeah maybe we are in a pre-scientific phase of stumbling around, shaking the tree indiscriminately and picking up the fruits. We don't know where exactly the fruit is, but we keep being fed nevertheless.
During the history of science, big discoveries always needed a preparatory pre-scientific phase of mere aggregation of experiences, stamp collection if you will, developing almost superstitious practices that may turn out to be false on the level of the individual components, but in aggregate they result in a net positive.
And it isn't guaranteed that the unreasonable effectiveness of math (a small number of simple and elegant formulas) also applies to this messy subject, as opposed to physics. Reality doesn't owe us any simple and satisfying explanations.
I don't think it helps to complain that we don't get knowledge delivered. The right reaction is awe and curiosity and marvel that a nonrigorous process has led to this. The detractors like Gary Marcus never foresaw this state of things and was sure that things would plateau earlier. But that in itself was a hypothesis about the crowd of monkeys that researchers are, and whether their output will be such or such. And it turned out to falsify the pessimistic views of many. In any honest interpretation of what the early pioneers like Turing set out to do in terms of the Turing test for example, we have made real strides forward in capability.
I know that you're eager for knowledge not just artifacts that work, but it's like complaining that a chess player won by making weird moves that are not listed in any textbooks and we have no idea why they worked. Whether we have a bearing on what exactly make our techniques work, they work.
Btw, lest this causes confusion, my part in the "explainable machine learning for robotics" project is to machine-learn autonomous behaviours for an underwater search-and-rescue robot. My specialty is not explainability, it's rather learning ability I'm interested in.
That out the way,
>> If you don't trust BLEU and don't trust my gut, how about a randomized survey of professional translators, or how many seconds per character a translator would need to fix up the raw output into something they'd be comfortable with calling accurate? Like, I'm not stuck on a metric.
No, sorry. I want a metric I can trust and not one that is equivalent to good, old-fashioned eyballing. If I wanted to eyball, I got eyballs, I can eyball. I want a metric that doesn't depend on someone's subjective evaluation. Unfortunately, the day that someone will come up with an objective test for machine translation is the day we'd have solved machine translation, and possibly all of AI, to boot.
Btw, I studied NLP at the time when Google Translate had just switched to Neural Machine Translation. I'm Greek, and I remember on the one hand the dithyrambic self-congratulations on Google blogs, and on the other hand how ridiculously crap Google Translate was in translating from and to Greek. And it still is, almost ten years later (My MSc was in 2014).
I also have a cluster of friends and acquaintances who are all professional translators and interpreters and I know that they are still ripped off by translation agencies who give them automatic translations claiming that they only need an hour of work to correct, when they need maybe a day or two. That kind of test will also not do.
"Progress" in machine learning means that every few years the hype peaks for yet another AI fountain of youth, or AI El Dorado, then that loses steam, and there's some sanity for a few years until the net thing comes along and trends on social media, then everybody starts writing super-serious papers about it sounding all business-like and scientifically minded, until Google, or OpenAI now, or some other big gun releases an implementation and everyone has to run behind them like idiots. It's basically science by twitter and it's completely bogus.
Bottom line, you want to make machine lerning work? Be Google.
>> The right reaction is awe and curiosity and marvel that a nonrigorous process has led to this.
To what? I think I've made myself very clear that I don't think we've got anywhere worth talking about. So my computer can now talk back to me. So what? Why is this a scientific achievement? Why does this answer scientific questions? Which ones?
>> During the history of science, big discoveries always needed a preparatory pre-scientific phase of mere aggregation of experiences, stamp collection if you will, developing almost superstitious practices that may turn out to be false on the level of the individual components, but in aggregate they result in a net positive.
I've heard this before but I don't know any examples where technological progress did not go hand-in-hand with scientific understanding. For example, when the Wright brothers made a machine that could fly, the followed on the footsteps of probably hundreds of others who had done the work of science, developing such necessary concepts as aerodynamics- like George Cayley, Otto Lilienthal of course, and in fact the Wrights themselves had set up a wind tunnel and studied the aerodynamics of their machines.
The difference with AI of course is that it's much easier to point to a thing and say "it flies". But that's another matter.
Lest I be misunderstood: I'm enjoying our exchange. Thanks.
I don't know exactly what kind of knowledge you want. Cybernetics and AI from the 40s and 50s onwards didn't aim at learning the laws of intelligence like the laws of gravitation, they were practical goals of building a thing. Like the Manhattan project wasn't about leaning stuff but about going kaboom.
Chomsky complains the same, that ChatGPT teaches us nothing about human language at all, it's not science, it's not advancing linguistics, but sure these things are good for captioning videos for those hard of hearing or translating between languages. And I'm like dude, nobody is stoked because they think we learned something about human language or the nature of intelligence. People are stoked because the computer can talk back at them.
And again I get where you are coming from, I see the same picture but see the rabbit instead of the duck. I see it as very important and consequential that these things work so well. I'm very big on eyeballing. I don't need to measure the voltages under the hood and stare at the speedometer to tell whether a car runs or not, may it run on electricity, natural gas or petrol, the wheels are rolling and we get from A to B.
Science isn't the outermost frame of existence, and has only been here for a few centuries. Journals and professional academia rat race of the shape as today, publish and perish etc, only since like half a century. The Romans built all their infrastructure without journals and h-indexes.
I get it, the hype often goes too far. But I'm now contrarian about the contrarianism. Denying that translation or transcription or image segmentation etc don't work is an increasingly silly hill to die on. So many have moved on to say stuff like yeah it works, but it's biased, it is wasteful and has a big carbon footprint, or can fail suddenly and unexpectedly out of distribution. But denial doesn't help.
I mean, the Internet wasn't a scientific discovery but TCP/IP, Ethernet and the Web still revolutionized society. Now maybe my grandma would deny this and say she gets by just fine gossiping with the neighbors and never using the Internet (though even her soap operas are delivered via IPTV). But just because it's not science, just "building stuff" it's still impactful. Don't be distracted by whether it fits a definition of science or not, that's a separate question from whether the thing works, by the definition of whether it gets jobs done that people want to get done and will pay for.
If you got jets and airliners, sure you can look up at the sky and see the thing working. But if you're at the stage where people are jumping off hills with makeshift wings tied to their arms, then you need some other way to know what works, and which way to keep digging. That other way, that is science.
This is a bit late now and you'll probably not see it, but it occurred to me that we are talking about different things, throughout. When you're talking about progress, you mean neural nets now work better than before in classifying images, recognising speech, translating text, etc. What I mean on the other hand is progress towards artificial intelligence. I don't have any difficulty accepting that image classifiers work better now than 20 years ago (although see my comments about why) and we perhaps don't necessarily need science to understand this.
What we do need science to understand is whether that kind of progress is progress towards artificial intelligence. And I would very much like to say that that is irrelevant because personally I don't care much about creating artificial intelligence, but a) that's the name of the field and b) many people in machine learning seem convinced that artificial intelligence is what they are trying to build (see e.g. how "the AI" became a common way to refer to neural net systems in recent years, even appearing in scholarly articles by people who should really know better, but, unfortunately, don't seem to). In other words, is progress in classification at all connected to progress towards artificial intelligence? That one is a scientific question.
The problem of course is that we have plenty of metrics to measure the performance of classifiers, but no good tests to detect the presence of anything like intelligence. I've thought of writing a paper to propose some but every time I'm like, nah, that's not what I'm working on, let someone else do it.
It's true that adherents to crypto have a faulty understanding of traditional finance just as the islanders have a faulty understanding of airplanes and canned food. But rather than painstakingly recreating the minute details of traditional finance in hopes of capturing the same success, crypto cults go out of their way to avoid recreating those details or understanding why they're necessary, if burdensome.