Comment by plaidfuji | Hacker News Reader

Comment by plaidfuji | Hacker News Reader

> “…the focus of AI in drug discovery must shift from doing what can be done - such as modelling data that is readily available, but that is unlikely to move the needle - to doing what should be done, even if this requires, for example, substantial data generation…” It’s a worthy goal, but I think that many involved in this work might be thinking, even unconsciously, “You first”.

This is the problem with AI for all of science - not just drug discovery. Applied ML has spread like wildfire through academia over the past decade - this started well before the LLM hype. It’s the perfect honey trap: research is painstaking and slow, ML offered a shortcut, and best of all, it just needs data. Research produces lots and lots of data! Surely this will be a match made in heaven.

I’ve watched the same pattern play out at least four or five times now in various roles.

(1) Propose an ML-guided approach to material/chemistry discovery/optimization.

(2) Gather existing data (real, experimental data).

(3) Realize there’s less than about 50 true rows of data on the outputs of interest.

At this point, you either: (4a) revert to traditional methods but keep the veneer of using ML to save face, or (4b) pivot to computational/simulation work or a high-throughput system that’s very far removed from your original problem, but allows you to keep playing with ML toys

It’s really bad. I left the industry. I don’t know how long it will take for people doing real science to take back the reins (and the funding).

The real value right now is in figuring out how to generate robust data cheaply and quickly. I'd wager that the effect of a good model on marginal data is small, but the effect of a marginal model on great data is probably quite large.

I completely agree with you. Data quality is the most important. One thing that's sad is how much academic data that's not reproducible. https://en.wikipedia.org/wiki/Replication_crisis

The obvious question is what limits getting more data? Astronomy (especially in Australia) has been quite good at designing surveys to answer multiple scientific questions with reasonable amounts of data (and then fed into ML systems like the cannon). Sadly one of the consequences of the LLM hype is the increasing cost of doing this, so "AI" is actually making things worse not better.

Here is an article by Pat Walters on the usefulness of ML in drug discovery. This article is a response to another one making the case that utility of ML models are very limited in drug discovery

https://patwalters.github.io/Response-to-Peter-Kenny/

> (4a) revert to traditional methods but keep the veneer of using ML to save face

I haven't worked in the industry side of things but in academia everyone kind of agrees that gradient boosting trees are some of the best models to do these things.

(3) seems like a problem in its own right? Basing science, traditional or newfangled ML, on such small amounts of data looks pretty weak.

In chemicals and materials, 50 rows of good data is a really solid study. That’s e.g. a 3x4x4 experimental design (assuming replicates for each condition get averaged into a single row). If you managed to prep that many samples correctly and obtain consistent characterization data across all properties of interest, you’ve easily got a paper. It’s also kind of malpractice to jam this type of data (few samples, wide rows) into modern ML models. There are plenty of simpler statistical methods that will tell you what’s going on, and even then a well-made plot might be good enough. The difficulty is not in drawing insight from the final numbers, it’s almost always in how those numbers came to be in the first place.

Thus the reticence of science-oriented companies to invest heavily in these mass data-gathering exercises to feed ML. It’s damn expensive, and almost always leads you back to raw data issues, not breakthrough discovery. Doing it without a set purpose in mind is even more likely to yield garbage.

I worked on some of the very best funded plant research out there. When it comes down to it, there's enough variation caused by confounding factors, and it takes so long to capture more data, that almost everything anyone tries cannot be called a success or a failure for years, because the individual measurements for one small plot of land somewhere just don't mean anything. Once you do an entire experiment for a season, which takes months, and you grab the little noisy data you have, and turn it into real rows, we were down to very little.

You can do more tests on smaller things, like checking if some protein will kill some cells of a pest, but making sure a plant produces it enough that it actually does something significant to the real, live pests, that it's not toxic, and it doesn't harm the plant's yield massively (as it's now spending time producing your pesticide) is still going to take years. We might be able to fold proteins, but the kind of things we'd need to really simulate plant biology well enough to not need years of failures are still very far away.

And it's far worse in medicine, as with plants at least nobody has ethical concerns if they fail and die, and nobody needs to get consent from a corn seed. Getting to 50 actual data points from many medical studies is already a lot of effort. And imagine when it's a long term study, and you need to follow patients for 30 years, as theym move, or die, or decide to stop participating, or who knows what.

Ethical concerns might become an even bigger bottleneck in the future. It's sad what we're doing with millions of rodents each year.

Yes indeed, I call it the revenge of statistics! Pseudoreplication rears its ugly head once again. Nonparametrics are certainly useful, but once you know things about the DGP, it’s often possible to do something “traditional” that’s better. Probabilistic programming, for life!

It was always easy to come up with new molecules/drugs/materials. The thing that changed is the scale that it can happen with the new ml-based approaches.

What hasn’t changed is finding ones that are manufacturable/synthesizable.

Even if you find 1 million new stable molecules, there no guarantee that even one of them is manufacturable.

Reply on news.ycombinator.com