Why Can't I Reproduce Their Results?
theorangeduck.com
theorangeduck.com
I worked in a cancer lab as an undergrad. New PI had just come over to the school. He had us working on a protocol he'd developed in his last lab to culture a specific type of glandular tissue in a specific way.
I and two other students spent months trying to recreate his results, at his behest. When we couldn't get it to work, he'd shrug and say some shit like "keep plugging away at the variables," or "you didn't pipette it right." I don't even know what the fuck the second one means.
Then an experienced grad student joined the lab. He spent, I don't know, a week at it? and was successfully reproducing the culture.
I still don't know how he did it. I just know that he wasn't carrying a magic wand, and the PI certainly wasn't perpetrating a fraud against himself. It was just, I guess, experience and skill.
A novice was trying to fix a broken Lisp machine by turning the power off and on.
Knight, seeing what the student was doing, spoke sternly: "You cannot fix a machine by just power-cycling it with no understanding of what is going wrong."
Knight turned the machine off and on.
The machine worked.
That is, I don't mean that I expect it to be a true story, but if you were to cite this parable, how would you cite it?
(setq stories (list "http://catb.org/~esr/jargon/html/koans.html"))
(car stories)I have a similar story involving myself and my sister. She was power cycling our family computer over and over, becoming more and more frustrated as it went into 'safe mode' equivalent. I allowed it to fully power on and then rebooted it, and it was 'fixed'.
The story isn't unreasonable.
Lab technique is definitely a thing. Mostly it's about knowing what not to pay attention to so that you can give extra attention to the things that matter. It's also about recognizing the intermediate signals that say "Yep, still good." (ie. Are the bubbles the right size? Is the foam the right color? Does it smell right/wrong? Did a layer of glop form at the correct step?)
However, I rarely see anyone taking good enough notes to reproduce an experiment. I had a Physics Lab taught by a professor who would randomly take someone's notebook in an assignment and reproduce the experiment using your notes--and fail you on that assignment if he couldn't.
Very few students ever passed that. You really have to be excessively meticulous. However, when you are debugging a real experiment without a known correct answer, if you aren't that meticulous you will never find your own bugs.
I have a textbook of cookery [1] written for students intending to become professional cooks, it emphasizes precision measurements and repeatable processes. Where it expects adjustments it explains what it means and what constitutes "it looks right". It stands head and shoulders above pretty much all other cookery books I have read. But of course it lacks glossy pictures and is not backed up by a video of Nigella Lawson in a low cut dress (not that I'm objecting to Nigella and her cleavage)
[1] Professional Cookery: The Process Approach by Daniel R. Stevenson
Emailing the original papers authors I managed to extract the comment "we place the lid lightly on top of the vial" (which only stood out in a photo as something which didn't make sense) and then filmed a bunch of my vials and noted that whether it worked depended on how the water evaporating/condensing sealed against the loose fitting lid.
(This was after a lot effort spent drying ethylene glycol too)
Turns out, you can get a perfect result everytime if you drop the reaction vial headspace oxygen to 17-19%.
People just don't monitor what they should (or take a 1/10 success rate since for most nanoparticulate research one vial goes a long way).
I'm still really proud of that but damn does it explain replication problems to me.
So, it really depends on what is being "reproduced". Is it the outcome? Or is it the process? If it is just the outcome, then the original authors could merely store an example of it as the only proof necessary. But, my intuition says that it is much more often the process that is needing to be reproduced/proven.
Therefore, "lab technique" should have nothing to do with a proof of process. Either it can or it cannot be done as it has been defined.
You can't define everything precisely--and inaccuracy may not lead to failure--just suboptimal results.
My favorite example of this was a semiconductor fabrication lab in college. There were 4 or 5 masks which have the step of "Align masks on lithography machine using alignment marks on silicon and expose photresist".
Okaaaaay. No big deal, you're putting lines and crosses in the center of other lines and crosses visually through a stereo microscope. Humans do that well.
Erm, that lithography machine is possibly older than the professor and likely hasn't been maintained in about the same timeframe.
So, the vernier screws have an enormous amount of backlash. If you've used shitty guitar tuners, you know how to deal with this. If you haven't, you spend a lot of time being frustrated figuring out what backlash is and how to deal with it. Okay, that's lab technique.
However, if even if you do know about backlash, this wastes time on a piece of equipment which is already time constrained. If you actually know something about semiconductors and think a bit about the process, you realize that Mask 2 is the crucial alignment because it defines your critical dimensions and screwing it up will haunt you while you can be a little more cavalier about the other masks. So, you spend way more time fighting the machine on Mask 2 to get it right (because it's important) and you spend less time on the rest. That's also lab technique.
Finally, if you're paranoid, you test the measurement equipment and calibrate it every single time you enter the lab. You note the serial numbers of the broken ones and only use the ones that reliably give correct measurements.
Your prize for this level of lab technique is that your transistor gives gorgeous graphs and works exactly like the lab claims it should while everybody else gets mushy results, at best.
The reflection hologram turned out to be the best one the lab supervisor had seen in a decade. I have no idea why it worked at all, let alone why it turned out so good!
Both processes are simple conceptually, but incredibly finicky in practice. The interference fringes can be ruined by wind outside the lab shaking the walls, which then shakes the optical table through the floor. You can't feel it, but the hologram is ruined. The laser could have a short coherence length, and you wouldn't know it. If it warms up too fast, it'll shift the fringes. Don't cough. Don't bump the table. Don't bang the door. So on, and so forth.
You made an important point in considering that other causal factors can be present besides fraud in science's reproduction problems. But, it appears like this was made to suggest that fraud is not as serious of a problem as it is in the reproduction crisis.
If more emphasis is placed in the scientific community on the necessity for proofs, then fraud would stand out like a sore thumb. Until then, it can easily be masked within the lax atmosphere of (a lack of) explicit reproduction specifications.
It does get laid bare eventually. I've heard of a well known PI being investigated by their funding agencies due to a post doc doctoring a western blot result. I heard a story about a grad student who tried to repeat a graduated grad student's protocol. They tried and tried for years without much success, initially blaming their own ignorance given they were a freshman when they started their efforts on the experiment. Eventually, after going back and forth with the former student, the PI, and collaborators, it was revealed the original protocol was outright wrong. In fact, more digging was done and that former students thesis research turned out to be fraudulent. The former student lost their degree and their current job, and the current student wasted two years of their life trying to make something impossible work. It's awful when fraud happens for everyone who is involved, whether intentionally or not. Fraud does tend to shake itself out, but it can take painful time.
TL;DR - I was being too careful, and so too slow. Because the binding was transient. So once I focused on speed instead of precision, the assay worked fine.
I've also found a few situations where I couldn't get somethign to work and it turned out the person who had it "working" wasn't running controls (positive or negative), so you didn't really know if it worked.
"I still don't know how he did it."
Practice. Learning through mistakes. Tips from others.
It was a matter of ejecting the pipette contents at the right moment while removing the pipette from the medium at the correct speed (I exaggerated my ignorance a little bit for rhetorical effect.) But there was no way to verbally teach the right "feel" for it - you just had to do it over and over until you learnt how pipetting "right" felt. You could (and I did) observe my PI do it the "right" way, but you can only get so close to hitting a baseball by watching a pro - at some point, you pick up the bat and learn by swinging.
Semantics. Familiarirty with a piece of equipment versus familiarity proper technique when using it.
Almost to the day I worked out the issue, I saw an immensely more experienced lab person running an almost identical experiment to my setup. They always added another reagent "in case there's contamination".
http://pages.cs.wisc.edu/~kovar/hall.html
"Electron Band Structure In Germanium, My Ass", is a great example of this... I'm pretty sure they had a bad contact (using indium solder?) when they ran the experiment. When you're getting garbage like that... stop and see if you can reproduce a single result, and until you can, something about your method/equipment/model sucks.
This sounds like a complicated way of saying the procedures for some scientific research are underspecified.
So, if you're in software enjoy the fact that you have the wonder of a compiler, an OS, and a CPU you can (mostly) trust. The rest of the physical world dreams of that level of control and repeatability for anything like the tiny incremental cost. Other than math and theoretical physics that's as close as you'll get to fully specified.
The way the host shows mistakes, troubleshooting processes, recovery attempts, and even the outright failures is so refreshing after endless canned presentations focused on the final success, ignoring years of trial and error.
Often, it's the trial and error part that's actually important, the end result not so much. The journey, not the destination, if you know what I mean.
https://www.youtube.com/channel/UCFhXFikryT4aFcLkLw2LBLA
PS: I actually did the semiconductor conductivity experiment at University, and it was a blast! It was supposed to be a 12-hour experiment, most of which was collecting the data by hand and then analysing the curve. Fitting two nonlinear functions using fancy graph paper sounded awful dull. Two hours of playing with liquid nitrogen followed by ten hours of paperwork? No thanks! Like the guy complaining in that blog, I swiped the (at the time) fancy digital oscilloscope from the senior lab, output the data to a floppy, and did a nonlinear curve fit in Mathematica. Because it could fit both regimes at once, I got about one digit more accuracy than the reference text provided. The professor gave me 6 out of 5 for my score, and I finished the 12 hour lab in 4 hours. It was my first real life lesson in the power of computers to solve physical problems.
Maddening.
When the right amount of info is shared a lot of cooperation is possible and challenging for both I suppose.
The added value for both is learning the other skill. Gaining a broader knowledge to engage further in career and life.
But I agree that the article is not good.
it should not depend on mindless/random trial-and-error guesswork and faith-based replication.
yes, it's hard to write everything down well. look at the recent DARPA paralell replication study. but it gives very high quality research/science for a little overhead.
Social sciences, where you're taking a bunch of data and running a Python, R, or Julia script against it? Sure. Require the PI to archive the data (or anonymized version where PII is an issue) along with the script and a quick description of the method.
When you cross over to more STEM oriented fields, tacit knowledge (e.g., the 'experience' or stuff you pick up from your colleagues) becomes much more important. Reproducibility is still important, but you can only expect the PI to provide so much knowledge before it becomes untenable.
Your comment is the equivalent of saying there should be no armies and instead we should be peaceful and love each other. Great idea. And those excuses of why I cannot connect A to B? Let's just reach consensus to use the same connector everywhere!
I'd like to live in that world, but unfortunately it is not the one we got.
It's not that difficult. There's already a Distill Prize for Clarity in Machine Learning. A great spark would be if a company like DeepMind or OpenAI would enforce standards like these internally and host a conference that rewards papers at that standard. It would be a great PR move, at great benefit for humanity.
I am not too familiar with the AI field, but I know that at least in my field the quality is very often too poor. We have to try to improve it. But if I rejected every paper I review because every detail to replicate it is not there, I would have rejected 99% of them. And some of the ones that eventually got accepted were very valid and very useful papers, and replication not so important after all.
Anyway it's more complicated than what you make it sound. For instance, if you don't like the culture at Google, you have a choice to not work for Google. And if you don't like the culture in some field of research you also have the option to not join that field of research. For sure, you don't know _exactly_ what you're getting into before you get into academia, but students who are interested in doing a PhD already have some experience -often, substantial experience- of how things go and even of the behaviour of the professors they choose as their advisers, with whom they may have done a Master's project etc. Some have publishe papers, presented posters and joined conferences already from their Master's or degree. And so on. Very few people just stumble into a PhD by mistake.
But that said, am I totally in the wrong on that?
Wiser now, I know I was wrong about the classes. As this article shows academic code experience does not necessarily mean anything in terms of code smells. That author sounds like how I used to code. I guess some never realized there are better ways, that their code doesn’t have to be so Byzantine and hacky.
But I guess part of the problems with "publish or perish" is that there is very little incentive to keep working on code after a paper is submitted.
Edit: So, I spent a few years in the industry before starting my PhD so code quality was important to me especially because my first proper job was in a shop that did code reviews and all. How did you get to appreciate the need for good coding standards? And how did you learn how to do it right? If you don't mind me prying! :)
Whether it's true or not, I don't know, but it wouldn't surprise me.
Preparing for the next paper?
Switching to Python and seeing enough python code helped too, but I think other widely used languages would have been about the same like JavaScript. Seeing a lot more code than just googled matlab snippets was a big part of it.
Anyway essentially it had become increasingly apparent that “there’s got to be a better way!”
It's not just a computer science problem btw -- it encompasses physics too.
Most people in my field use either IDL or Python (if they're doing data analysis) or C/Fortran (for heavy-duty simulations). Some also know Matlab or Mathematica. I've been trying to convert people to Julia -- it's gonna supplant Python for data analysis and maybe even C/Fortran for simulations.
Much of the recent "efficient" DL work is like that. Efficient models are notoriously difficult to train, and all manner of secret sauce is simply not mentioned, and without it you won't get the same result. At higher levels of precision, a single percentage point of a metric can mean 10% increase in error rate, so this is not negligible.
To the authors' credit though, a lot of this work does get released in full source code form, so even if you can't achieve the same result on your own hardware, you can at least test the results using the provided weights, and see that they _are_ in fact achievable.
In fact, I really do not understand why in 2020 we are writing papers/PDFs anymore. We have more than enough software and software tooling to have superior formats for sharing and discussing information.
Is our biggest hurdle to this that the most experienced scientists/professors/researchers in our day are not as technically competent as they would need to be? Otherwise, why is this not already the norm? We do not need ArXiv or journals, we need GitXiv.
1. Researchers ditching papers as the primary means of exchanging ideas.
2. Researchers going back and fixing bugs, or even reviewing PRs.
None of this is going to happen.
- assuming highly specialized Math skills (e.g. manifolds), which wasn't me at the time, so it made testing and bug hunting harder
- code missing, or code is a long C file vomited out by Matlab, with some pieces switched to (Desktop) SSE instructions for speed gains
- papers were missing vital parameters to reproduce the experiment (e.g. exact setting for a variable that influences an optimization loop precision vs speed)
- the experiment was very constrained and the whole algorithm would have never worked in real life, which is what I had supposed for the first few months (meh)
- most papers, as the article says, are just a little bump over the previous ones, so now you have to read and implement a tree of papers
- sometimes there would be the need of a dataset to train a model, but the dataset is closed source and incredibly expensive, so not a good avenue
At the time I was also working with the first version of (Apple) Metal which, after going crazy on why my algo wasn't working, I discovered had a precision bug with the division operator. FML
Still it was a very instructing experience, the biggest takeaway if you do something similar: don't be certain that once you implemented an algorithm, it will work as advertised. It's totally different from say, writing an API, it's not a well constrained problem.
This has been my experience with a large number of papers that make all sorts of wild claims. Whilst technically true they are more often than not are not actually very useful for any practical application.
How do they manage to push this non-portable waste of time on people?
Obviously they can push it because of the platform. But developing with it is not very different from CUDA/OpenCL.
Yes I agree, nine times out of ten you've made a typo, connected the red wire where the black wire should be.
But I diagree with the overall sentiment of the article which fails to highlight that - when all else has failed and you have double checked things your end ..... sometimes there IS a typo, an incorrect equation, a simple error IN THE ORIGINAL research. Sure, blame yourself as a first instinct (which is a good thing to do) BUT - There is indeed a replication crisis currently in all fields of study and research. Forgive the link to a totaly non-tech field but it illustrates my counter-point as well as any other.
"The Role of Replication (The Replication Crisis in Psychology)"[1]
The openness of psychological research – sharing our methodology and data via publication – is a key to the effectiveness of the scientific method. This allows other psychologists to know exactly how data were gathered, and it allows them to potentially use the same methods to test new hypotheses.
In an ideal world, this openness also allows other researchers to check whether a study was valid by replication – essentially, using the same methods to see if they yield the same results. The ability to replicate allows us to hold researchers accountable for their work.
[1] https://courses.lumenlearning.com/ivytech-psychology1/chapte...
Tight regulations on reproducibility are the first thing academia needs. Academia is a rat race with so much dishonesty and optimizing for measure (citation count etc..) these days. I have seen too many of professors producing low quality papers for the sake of producing papers - which creates a lot of noise, and a cargo cult PhD culture. Without reproducibility how can you even trust the results?
While academic code doesn't adhere to code quality standards of software development, I don't think many software engineers involve in ridiculing the author if they produce the code, leave alone other academics.
Btw, I submitted a link (a very comprehensive article by a CS academic) but it was lost in the HN noise. Maybe someone with higher karma points repost that? I found that article through nofreeview-noreview manifesto website and that's very well written, it covers a number of problems like this.. (http://a3nm.net/work/research/wrong).. PS I am no way affiliated to the author. Just mentioning because that article deserves to be at HN front page.
Irreproducibile results in computing have no justification in this day and age.
That's like saying you can't learn from exercises in the book if the answers are in the back.
I ran into exactly this problem with:
White's Reality Check (WRC), described in White's (2000) Econometrica paper titled "A Reality Check for Data Snooping",
Once we got there for what we could we found two fundamental problems with "Whites Reality Check", one theoretical that is blindingly obvious once you see it and the other that they ignored trading costs. When we measured the levels of trading costs that would permit the algorithms to work it was clear they could not.
That we could not deduce without reproducing the results. If we could have copied the original code it would have shaved about eight weeks from my research.
BTW "Whites Reality Check" was at the time pone of the most cited paper in its, rather narrow, field.
You suspected the result was wrong, so your interest was in debugging their code to contradict their result. That has value but also is not a reproduction of the study.
Usually, the highly specific experience and knowledge of the author is assumed in the reader...
We've built the platform from the ground up with immutability in mind, leveraging Clojure and Datomic which are a great fit for this architecture.
I'm waiting for the textbook that offers formulae with code and a but of regression data.
[1] https://en.wikipedia.org/wiki/Robert_Recorde
[2] https://www.cs.utexas.edu/users/EWD/transcriptions/EWD10xx/E...
Obviously some cutting edge stuff can't use standardized equipment, but can you standardize a lot of other stuff?
I obviously don't have the experience to enumerate what is important. haha but maybe its still a worthwhile idea
I'm currently trying to push the use of an existing standard [1] for capturing engineering experimental results.
If this is the case in a majority of the instances where someone fails to reproduce the software backing a paper then there may be another issue at play. Someone re-implementing a paper is "just" taking a written description of something and transcribing it to code. If a mistake, that can completely ruin the results of the work, can be made this easily it should be fairly straight forward to see that the original implementer could also have made a mistake that could have thrown off their results.
Does the world of academia have any tools to prevent this? It seems like this could effect a lot of the research being done today. Given the following:
1. Most research being done today utilizes some software to generate their results.
2. This software often encodes some novel way of implementing some analysis method.
3. The published paper from research with a novel method of analysis will need to describe their method of analysis.
4. Future researchers will find this paper, attempt to implement the described analysis method, and publish new results from this implementation.
We can see we are wasting a lot of resources reimplementing already written code. We may also not be implementing this code correctly and may be skewing results in different ways.
> Debugging research code is extremely difficult, and requires a different mind set - the attention to detail you need to adopt will be beyond anything you've done before. With research code, and numerical or data-driven code in particular, bugs will not manifest themselves in crashes. Often bugged code will not only run, but produce some kind of broken result. It is up to you to validate that every line of code you write is correct. No one is going to do that for you. And yes, sometimes that means doing things the hard way: examining data by hand, and staring at lines of code one by one until you spot what is wrong.
This is a very common mindset that many academics have but I don't understand why this is the case. A paper seems like a fantastic opportunity to define an interface boundary. If your paper describes a new method to take A and transform it into B then it should be possible for you to write and publish your `a_to_b` method aside your paper. You could even write unit tests on your `a_to_b`. If a future researcher comes along and finds a way to apply your `a_to_b` to more things they could modify your code and, just by rerunning your tests, should be able to verify their implementation actually works.
If a future researcher decided to use you `a_to_b` you could write some code to automatically generate a list of papers to reference.
If we are seriously spending this much time treading the same water then it should be possible to dramatically improve the quality and throughput of academics by providing some tool like this to them.
I know someone will say "but you gain so much knowledge re-implementing XYZ" and to them I'd say that you don't need to read the code while reading the paper. You could write the code yourself and just utilize the unit tests provided by the author to make sure you fully understand each edge case.
Publishing code in a journal article in analogous to publishing key equations. Equations are just functions themselves after all. However, articles have to be succinct, and journals still exist in a hybrid physical/digital space. It isn't very useful to physically ink thousands of lines of code onto a page, especially if it represents a vastly different level of technical detail than the rest of the paper. In practice, people who write comprehensive software typically make it available through github or similar, and put a reference in the article. If not that, people will send you their code if you contact them. If they're stingy you may not get source, though. I don't know of any funding agency requirements that code source be made available, though I think that might be a good thing to try.
I think the biggest difficulty is with medium-sized pieces of code. Small enough that the bookkeeping/maintenance needed to make it available gets skipped, but large enough that it isn't possible to provide details in the article.
BACKGROUND I have been working for 15 years in industry doing hardcore ML. I have the fortunate drive and background that I was able to get my masters degree while working full time from an R1 school. No watered-down online degree, no certificate. I would drive twice a week to class for 4 years and did a full thesis which was published. Since then, I have published 6 papers, all peer reviewed. I even did a sabbatical with another research lab of which I was invited to come.
After 15 years, I decided to go back and get my Phd, all while continue to work full time. My thought was that it would be easy to get a phd with all my technical experience and math chops ive developed over the last 15 years. I essentially have been doing math 5 days a week for 15 years. Here's what happened...
Coursework was a breeze. I barely put any time into it and I easily can get a B+. This is really helpful because I am working 40-50 hours a week at my full time job nd managing my family. I passed my candidacy exam on the first try with little issue (this is rare for my department).
The biggest hangup I have about the phd process is what my advisor wants me to do when writing papers. He is the youngest full professor in the department and is from a well known and well respected graduate research university. But, the way he has me slant my papers is absurd. Results which I feel are very important to the assessment of the reader to decide if they should use the method, he has me remove because the results are "too subtle." He is constantly beating on me to think about "the casual reviewer."
Students in the lab produce papers which are very brittle and overfit to the test data. His lab uses the same dataset paper after paper. My advisor was so proud of a method his top student produced that he offered his code for my workplace to use. It didn't work as well as a much simpler method we used. Eventually we gave the student our data so there could be a fairest shake at getting students method to work. The student never got the method work to work nearly as well as in his published paper despite telling my company and I over and over that "it will work". The student is now at amazon lab 126.
STATEMENT: Academia is peer reviewed driven but the peers are other academics and so the system of innovation is dead; academics have very little understanding of what actually works in practice. Great example: its of no surprise that Google has such a hard time using ML on MRI datasets. The groups working on this are made up of Phds from my grad lab!
TL;DR - worked for 15 years, went back for phd, here's what i hear:
"think of the casual reviewer"
"fiddle with your net so that your results are better than X"
"you have to sell your results so that its clear your method has merit"
"can you get me results that are 1% better? use the tricks from blog Y"
"As long as your results are 1% better, you are fine"
Edit 1:boasts are given to avoid "your experience doesn't count because you X" strawmans, where X={are lazy,are a young student, are inexperienced, went to an easy school, are in an easy program, naive to the peer review process}
When I was doing my PhD (Statistics) I spent several weeks running simulation studies to compare our new statistical model to 2-3 existing models that did similar things, across a wide range of data sets both real and generated with varying parameters. It was an unsupervised learning problem so there wasn't a "correct" answer to compare to on the real data sets, but on the simulated data we outperformed the existing models over about 60% of the parameter space that we tested, including the part of the parameter space that I thought was most useful/likely in practice.
Rather than present all the simulation results and have a nuanced discussion of where we did better and where we did worse, my advisor had me remove the 40% of the parameter space where we did worse from the results, so it looked like we always did as good or better.
The scientific contribution would undoubtably have been better if we had presented the results where we did worse and discussed why we believed that to be the case, but getting the paper past reviewers and into a top journal (as we eventually did) was considered more important.
In a very large number of cases, the startup closing its doors after two years is not considered a failure by the VC fund. The startup successfully spent the money the VC firm was contractually obligated to invest, may have employed the VC's choices of officers for a significant period (providing them income and experience), maybe purchased a great deal of tech from suppliers the VC officers are themselves invested in, and may have left valuable assets that could be snapped up.
The biggest problem a VC firm has is the five billion dollars of others' money they have to "place" in 30/60/90 days. What happens after placement is much less their problem. They know most of the placements are duds, but they and the actual investors knew that up front. Once the money is "placed", though, much of it can be siphoned off for the benefit of the VCs' cronies or one or other non-dud. Maybe a non-dud or non-startup buys up assets of a dud for pennies on the dollar, and extracts something usable, like patents or equipment. Sure, the investor lost that money, but somebody got it, and somebody got what it bought.
None of this is good for most people who do a startup, unless they are chosen as a non-dud. The chosen duds are valuable for money laundering, which few startup principals really meant to sign up to be. Some did.
Not always, it's of course in the VC's interest to push the company forward and secure further VC rounds. You could have a failed initial product but play the "product market fit" or Pivot game essentially until you run out of money or find something that sticks. You can easily raise money as an AI startup with lots of hype only to be shipping a half baked marketing platform a few years later and still raise more money from VCs.
https://www.fda.gov/about-fda/center-drug-evaluation-and-res...