AlphaFold won’t revolutionise drug discovery
chemistryworld.com
chemistryworld.com
What the scientists (all from Roche) said was, more or less, "yeah, that helps a lot. It doesn't solve the whole problem, though."
20 years later they've gotten yet more help with Alphafold. Once again, they can do things faster, but it isn't a Moore's Law-type of change. It's still a really hard problem demanding culture, animal, and human tests, and those take time and money.
It's absolutely a step change in demonstrating what's possible. The author can sit and criticize alpha fold from his armchair, but the dude is missing the forest for the trees.
Do you remember how people on HN used to take comfort that AI couldn't write code? Now they're taking comfort in the fact that AI can't write code well.
We're headed towards a word where data gets put in a blender to decide what to try next. That entire pipeline can and will be automated. And in that world, human ingenuity and expertise may become far, far less relevant.
Let's hope we'll be dead by then...
The point is a great deal of drug development time did not go away when DNA sequencing and bioinformatics became available, and a great deal will not go away with Alphafold.
> We're headed towards a word where data gets put in a blender to decide what to try next. That entire pipeline can and will be automated.
No, we're not, and no, it will not. Alphabet has an entire large organization trying to bring the benefits of Big Data to traditional sciences. It's not a waste of time by any means. It's also not a steamroller ready to crush every problem in its path.
That is not particularly convincing, given that they've crushed multiple problems that were viewed as being decades away in just a few years.
It wasn't meant to "convince." It's an assertion.
So, it's going to land humans on Mars in the 2020s, too? How about cheap nuclear fusion?
Some problems are amenable to AI approaches, including many that were thought not to be.
Some are not.
"contributing to figuring out how" is not the same as "crushing the problem." It's helpful, of course.
There’s no incentive for the state to fund it so Musk has to do it while still relying on government funding for other SpaceX projects. If we get the National Security elites on board we’ll be there so fast it’ll make your head spin. We could have AGI 2-5 years sooner than we otherwise will (2030) if the government wasn’t run by geriatric humanities majors. These problems are mosyly downsyream from money, lots of it, and you need Manhattan project scale budgets for them.
AGI and fusion also require some theoretical breakthroughs but Mars? Mars doesn’t. Making it livable is hard but just landing there just means scaling up the moon landing.
Deep learning did turn out to be a very major advancement! And it's still bearing new fruit. But it does not at all imply that future advancement is inevitable or that advancement must continue at its present rate.
It's also worth considering that the domains where deep learning now excels are the same domains where it excelled when it first became popular (AlphaGo and AlphaFold are perhaps exceptions) and that all this advancement comes from hammering away at the same problem spaces for so many years, with huge budgets. Maybe the next area for advancement is something like combining fast on-device deep learning inference with robotics, and maybe that opens up a whole new world of possibilities. But that's a maybe.
What are these multiple problems you speak of?
No, actually: I am 40, and I don't remember that ever being the case. We could have computers write code back 20 years ago, at least... we've had automated scaffolding, pattern extrapolation, contract generation from test cases, random forests designed to approximate unknown algorithms; the code just wasn't good or useful or some other non-boolean axis, and that is exactly the debate we are having today, despite some clear expansions in the scope of practical applicability.
[0] https://www.joelonsoftware.com/2000/04/06/things-you-should-...
It is not true that AlphaFold team did not have domain expertise. I would accept that AlphaFold team had less domain expertise than expected. Could it be done with no domain expertise? Maybe, maybe not, but AlphaFold is not an evidence either way.
A lot of faculty shocked when AlphaFold 1 was released. AlphaFold 2 /really/ hit hard because the model used was even more "dumb" in that in leveraged even less domain knowledge in it's construction.
This was posted four years ago about a precursor of Copilot, there is no comment.
I personally consider this to be lazy and dangerous thinking. I agree that something like this is going to become increasingly possible for more and more scenarios, but in no way should "putting data in a blender" without understanding a good long term strategy for research and engineering.
The danger of losing fundamental understanding and alternative approaches because there is a low-effort sometimes-good-enough way is real and should not be treated lightly, much less cheered on.
The technique is what mattered. Not the experience. There have been many labs dedicated to this problem for years. They never even came close.
The first and corresponding author on the original Nature paper, Andrew Senior, was not a few years out of grad school[1], he's worked at Google for the last 14 years, and started working at IBM in 1994 [2].
[1] http://www.andrewsenior.com/technical/ [2] https://research.google/people/author37792/
Human ingenuity produces the data and builds the blender.
If you're referring to neural program synthesis with large language models (LLMs), the performance of current systems is pitifully low, with best results in the range of 5-7% correct programs when evaluating on a test set similar to the one used for training (and allowing a single "guess"; it goes up to 30% with 100 guesses; happy to point to references if required).
So no, "AI" (meaning deep learning I presume) cannot write code very "well".
Of course there exists an entire field of research in program synthesis that predates LLM code generators by a long time, and that can do much better than LLMs, but you haven't heard of it because DeepMind and OpenAI don't choose to champion it. But that's a story for another time.
[Correction: dammit, I misquoted the metrics. OpenAI claims ~30% correct results for their Codex model when allowed one guess on their test set; ~70% when allowed 100 guesses. See link to Codex paper in child comment. The 5-7% is the rate of correct programs for SalesForce's CodeRL on the "Introductory" level problems of the third-party APPS dataset and when allowed one guess; see Table 1b in the CodeRL paper summarising results on APPS for various LLM code generators: https://arxiv.org/abs/2207.01780]
On HumanEval, a new evaluation set we release to measure functional correctness for synthesizing programs from docstrings, our model solves 28.8% of the problems, while GPT-3 solves 0% and GPT-J solves 11.4%.
https://arxiv.org/abs/2107.03374
The numbers I quote (5-7%) are on the APPS dataset, reported by SalesForce:
https://arxiv.org/abs/2207.01780
(Recently shared on HN).
Note that there is no standard dataset like ImageNet or MNIST in program synthesis so it's not correct to speak of a "benchmark". LLM code generators in particular tend to be evaluated on whatever dataset their creators thought best demonstrates their system's capabilities.
But is there a distinction between drug discovery and drug development? Because in my totally uninformed brain AlphaFold could contribute to the former, but not the latter, which seems to be the focus of this sentence.
What do you mean?
Saw your reply too late to edit.
I don’t know how you got that link, but, to avoid that issue, use URL pointed at by the “X days ago” link.
> when your entire computational technique is built on finding analogies to known structures, what can you do when there’s no structure to compare to
Lots of people seem focused on the idea that deep networks can't do anything novel and are just like fancy search engines that find a similar example and copy it. This is not true. They do learn from much deeper low level structures in the domain they are exposed to. They can be aware of implicit correlations and constraints that are totally outside what may be recognised in the scientific understanding. Hence AlphaFold is quite capable of predicting a structure for which there is no previous direct "analogy". As long as the protein has to follow the laws of physics then AlphaFold as at least a basis to work from in successfully predicting the structure.
> It is very, very rare for knowledge of a protein’s structure to be any sort of rate-limiting step in a drug discovery project!
This and the following text are very reductive. It's like saying, back in 1945 that nuclear weapons would not be any sort of advantage in WW2 because it is very rare for weapons of mass destruction to win a war. Well yes it was rare, because they didn't exist. And so too did we not have a meaningfully accurate way to predict protein structures until AlphaFold. We've barely even begun to exploit the possible new opportunities for how to use that. And people have barely scratched the surface in adapting AlphaFold to tackle the related challenges downstream from straight up structure prediction. Predicting formation of complexes and interactions is the obvious next step and it's exactly what people are doing.
It's not to say that it will revolutionise drug development, but the author's argument here is that he is confident it will not and he really doesn't assert much evidence of that.
And what is your second criticism exactly? The author comes from the drug discovery industry. What the author said should be generalized to: even if we know the perfect experimentally confirmed 1A resolution structure of every protein out there tomorrow, that won’t exactly revolutionize drug discovery. That’s because protein structure gives maybe 10% of the context you need to successfully design a drug. It’s dynamics, higher order interaction specifics, complex interplay in signaling pathways in particular cells in particular contexts and what entire cells and organ systems in THAT PARTICULAR ORGANISM do when this protein is perturbed, are what truly affects drug discovery.
If you absolutely want to revolutionize DD, find us a better model to test things on that’s closer to the human body as a whole. Currently mice and rats are used and they’re not cutting it anymore.
This fundamentally goes back to the downfall of the prototypical math or software guy trying to come and say “im gonna cure cancer with MATH!” No you’re not. You’re gonna help, and it’s appreciated, but if you’re gonna truly cure cancer you better start stomping on a few thousand mice and maybe also get an MD.
In addition you make the claim [They can be aware of implicit correlations and constraints that are totally outside what may be recognised in the scientific understanding] and again this is not awareness in any way. Is is still a result of the constraints of the programming that goes into them.
Being able to predict something is not learning nor is it awareness. Mathematical equations predict things that can occur and these are not aware nor are the equations learning. The programming that underlies these systems is mathematics and as such are just as constrained.
I have said this elsewhere, all such systems are Artificial Stupidity systems. It takes a human mind (which we still do not understand) to look at the results. To say that these systems are more than they are is missing the usefulness that can be obtained from them. Irrespective of any claims otherwise, all such systems are simple and are in reality GIGO (garbage in, garbage out) systems. To forget this is to walk a path that is dangerous.
I have, over decades, had to deal with systems that appeared to give the "right" or "reasonable" outputs and yet when analysed where found to be in great need of redevelopment or in some cases thrown away as pure garbage.
Computer systems are useful but don't depend on them without human checking and control.
>They can be aware of implicit correlations and constraints that are totally outside what may be recognised in the scientific understanding. Hence AlphaFold is quite capable of predicting a structure for which there is no previous direct "analogy".
Yes, while an expert of their field, I suspect author doesn't have full understanding of what neural networks or ML is capable of, in this case it's not necessary to have seen fully similar molecules but it is sufficient for AlphaFold to have encountered basic blocks of the vocabulary that are familiar. One can see how this argument would be silly if (hyperbolizing) someone said that just because we are located in a tiny part of observable universe the physics we know to work here wouldn't work in Alpha Centauri. To make this point simpler, this is no different than suggesting that a model will break if it encounters an out-of-vocabulary word, whereas in reality a simple tokenization technique would ensure that in majority of these edge cases, the unknown word would be broken down into subwords, which, in turn, would still be familiar to the model and statistics that model has learned would still give it a reasonable guess as to what the sum of the parts entail. There was a recent work (https://arxiv.org/abs/2207.05221, in LLM, not molecular prediction) where models could be shown to "know" (oh G-d how tiresome it is to make these disclaimers, but models are NOT sentient) fairly decently what they don't know. Thus, I wouldn't be surprised if AlphaFold could at least give a confidence score to its prediction of folded protein, helping scientists who use it as a TOOL (which it is, it's not a solution) to exercise caution.
>As long as the protein has to follow the laws of physics then AlphaFold as at least a basis to work from in successfully predicting the structure.
This is pure speculation. There is no guarantee AT ALL that AlphaFold follows laws of physics other than the way predictions are clipped to be reasonable distances between molecules or other "hacks" authors have added as "inductive" (actually symbolic) biases to the model.
I agree with you, but just to point out that isn't what I wrote. It's not a speculation that it has "at least a basis to work from". Whether it actually does or not is speculative. The assertion in the article is that it has no basis to do it.
somebody bet big on AlphaFold and OpenAI but realize they are not going to break even at least for another decade or two.
Do you have a source for either or both of these claims?
https://www.nature.com/articles/s41592-021-01362-6
And the January paper from this team claimed 66%:
https://www.nature.com/articles/s41592-021-01362-6
This is impressive, but not game changing.
Solving protein folding is analogous to solving the halting problem, but harder. The sequence is interpreted by squirting out a string of stuff that mutually attracts and repels itself and the environment. Different temperatures, neighboring proteins (the "program input"), etc, leads to different folds (the "program output"). As with software, there's a happy space of expected inputs where it's hopefully easy to reason about expected outputs.
The difference between folding and the halting problem is that each step of the execution trace you'd need to generate to show a particular structure is reachable is quadratic in the number of atoms in the protein and surrounding environment, and the steps are not discrete.
Alpha fold mostly just glues together previously seen folds that were produced via x-ray crystallography. This technique is called "threading" and is decades old. It has all the obvious limitations you'd expect if you extended my halting problem analogy.
Also, most of the low hanging fruit in this field was picked a long time ago, so the 33% they perform poorly on is precisely the interesting part of the problem space.
Edit: Here's an early paper from 1997: https://pubmed.ncbi.nlm.nih.gov/9237912/
For whole structure threading, they get the correct fold "45-75% of the time". "Correct fold" is a weaker criteria than Alpha Fold uses in their abstract. However, it's interesting that Alpha Fold is still stuck in that range 25 years later. (Also, the 1997 paper is nowhere near state of the art.)
Producing a list of biologically common folds of a protein sequence without access to the actual environment they see is directly analogous to producing a list of all reachable statements in a program over all possible inputs.
Biological processes are analogous to running a program once on a well tested input.
Prion-based diseases are like fuzz or pen testing the folding process.
Of course, the mathematical formalism is a more useful way to think about computing (and I argue, protein folding).
Treat this as it is. An exciting approach that may help some now and yield fantastic results in the future. Don't count the chickens before they hatch.
Even if the structures were entirely correct - and they're definitely not - there's a massive complex metabolome to figure out.
Google is certainly milking the PR as much as they can, and that can be dangerous to the laymen approving research budgets.
Saying that AlphaFold makes x-ray crystallography useless is like saying DALL-E makes photography useless or Copilot makes GitHub useless. You've got the dependency chain backwards.
[1] (Or at least, they're treated as the ground truth---they don't necessarily predict the conformation of proteins in solution, but that's a separate topic for another thread).
Without machine learning at every atom is dynamic precision we are at 10^-18 L (liters) at 20 micro seconds a week with a specialised super computer https://dl.acm.org/doi/abs/10.1145/3458817.3487397 .
A solution does not need that precision everywhere. However a machine learning proxy of such precision in every relevant environment including 2d surface along non mixing fluid etc for every likely type of interaction is required so we can be certain of the possible outcomes.
That would allow humanity to pre-screen a bunch of edge conditions and check for unintended or previously explained side effects. The derived surrogates for environment dependent reaction rates could be used in a spatially distributed event based simulations with level of precision ranging from atoms with position and electrons in orbits subject to electro-magnetic force interaction, molecules as things with position and rotation and folding state, concentration gradients of those as stochastic 3d PDEs, 2d PDEs, 1d PDEs and ODEs of the number of moles with relevant boundary conditions. If we had those reaction rates down and knew of all the proteins and other structures i am positive that a proxy model of relevant parts of the human body could achieve enough accuracy to be practical at pre-screening drugs with todays super computers.
Unless of course you think you can truly model the entire human body pharmacologically using AI or whatever. That’ll help.
https://www.jax.org/news-and-insights/jax-blog/2020/july/why...
I suspect that, like any ML problem, this one is a small part of the whole solution of drug discovery. There are always system-level dynamics at play.
To me, some relevant questions before deciding to take on an ML problem tend to be:
[x] Does solving it eliminate manual labor from the process? [x] Does it save $ in the progress towards solving the whole problem? [x] Is it fun to solve it?
It’s kind of like anything in research, lots of small steps enable revolutionary breakthroughs every so often.
Of course there are plenty of drugs where we either don't know where they bind or we're probably wrong about where we think they bind. Or they bind at multiple places and some desirable or non-desirable effect are due to binding at places we don't know yet.
There are real uses to having lots of high-quality structure predictions for proteins. Drug development is something that only get limited benefits here. If you want to know how drugs or drug candidates bind to proteins you first create a protein structure with X-ray crystallography. Then you soak your crystals with your drugs or drug candidates and determine even more structures. The interesting part here is not necessarily the overall fold of the protein (which is mostly what AlphaFold gives you) but e.g. a single hydrogen bond to the drug in the active pocket of the target protein. You need really high-quality data if you want to do any kind of rational drug design, most of the time we still just semi-randomly vary structures until they bind better as far as I understand.
Given the utter dominance of Google advertising, I think the hating is a necessary counter in order to at least place it in its right place.
Whatever skill Google has computationally is more than matched by their media dominance and public relations prowess.
I don't like hype or hate that's devoid of nuance. But actual scientists working in these fields don't generally pay attention to these things as much as we might. They read the papers, and they have years of training to help them decide what is overhyped and what isn't. I'm not sure what happens on HN or in advertising channels has such a huge bearing on this.
The scientists are not the ones who need the counter-marketing, that's for the people who are not experts and only hear this one achievement (which is significant!) being trumpeted repeatedly as Google maximizes the PR benefits of conducting research.
It's like IBM's Watson play, except that there is at least a some serious meat behind AlphaFold.
How do we know the model hasn’t figured out some of the ‘whys’ somewhere in there?
But there's a reasonable intuition suggesting that the answer to your question is "no". What we're looking at is a non-linear regression model reproducing the function (which according to the article isn't really a function, but that's above both my and the model's knowledge) from a gene sequence to a 3d structure. It is heavily meta-optimised, so the "why's" would only be in the model, if reproducing the process of folding the protein was the cheapest way to guess the structure (). Intuitively it introduces at least one extra dimension, so should be way more expensive than finding analogues among known sub-aspects of the function. Hence, I would expect none of the "why's" to be in there.
Sadly, if any insight for the "why's" was there after all, we don't have a method to extract it anyway.
Disclaimer: I work in Google, far away from DeepMind, have no internal knowledge on this.
This has been my central frustration with working in ML. People always expect a "why" to exist, and by why I mean a cogent narrative explanation to complex phenomena. Maybe there is no "why" like this for a bunch of physical phenomena, maybe it is just a bunch of low level intricate stuff interacting in complex ways. There might be an emergent model that you can get a useful predictive model for with an ML model, then people get mad because the prediction doesn't solve the real meta problem that they were expecting to solve via the sub problem (e.g. solve folding then get mad because folding itself turns out not to be super useful because we don't know which protein to target, solve image classification then get mad because that doesn't make it easy to make a self driving car, etc, etc). "More is different" is definitely an idea in physics that needs to propagate into other fields to temper our expectations.
Why did does a cube on a inclined plane start to slide? You act like the correct answer is "because the subatomic particles and space time in the light cone of the experiment made it that way" when one should expect "because the sin of the incline angle times mass times local gravity became bigger then the static friction between a cos(incline angle) times the original cube weight and the surface at no incline" which is a lot simpler.
What's encouraging is the rate of progress, not what has already been done.
We do not have well-formed theories of the molecular details of many diseases. There is no immediate computational approach that address this defect. The community has had fairly simplified models for some time, and there's a lot of historical belief that by knowing protein structures in details, we can understand the nature of a disease through its molecular etiology, and from that, we can make drugs (either small molecules or biomolecules) that modulate proteins in rational ways to eliminate the disease with a minimum of side effects.
In my mind, much of the problem is similar to modern deep learning compared to previous techniques. Several extremely challenging problems (high accuracy voice recognition, image recognition, object detection) simple were not solvable through the statistical techniques and mental models adopted by the practitioners. It is not abundantly obvious that stupidly simple deep networks can be pretrained on enormous amounts of labelled data, or even unlabelled data, but we didn't even have the ability to know this confidently until we had the right network architectures, enough high quality labelled data, and adequate compute power to train them.
I believe that by starting to think about disease modelling from the same mindset as deep learning (simple models with many parameters, the models don't actually represent the assumed mechanism, large amounts of high quality data, lots of CPU, GPU, and RAM) and also thinking of the disease treatment process in the same way will greatly increase our ability to "understand" and "treat" diseases, while knowing far less about their underlying mechanism that we thought.
A common example is disease/patient stratification. If you've developed drugs that treat disease A, but it turns out later, there are really two diseases, A1 and A2 with different underlying mechanisms but superficially similar exterior symptoms, you'll realize why some percentage of your population didn't get better (and often got worse, given the underlying toxicity of some medicines). If we could just stratify diseases better, and classify patients into the right bins, the effectiveness of medicine will go up (and drugs will get through clinical trials faster/better).
None of this addresses the later-stage issues, such as successfully running all the phases of a clinical trial and the other gauntlets you must pass in order to get a drug FDA-approved.
I would continue to expect marginal improvements for the foreseeable future. But be aware: some companies already have managed to do a good enough job developing new medicines that they routinely create multi-billion-dollar blockbuster drugs year after year after year (my employer, Genentech, is a perfect example of that). It maintains an enormous and well-funded R&D arm that expends untold neurons attempting to understanding disease better even before we start to consider something as "druggable".
And if so, was this said about those projects? If not, why not?
Because the two ‶they″ in your sentence are different people.
Sure, for fundamental researchers, solving protein folding is a big thing – and although AlphaFold made strides in this direction, it didn't ‶solve″ it on any metric.
But the other they, DD guys, already knew for decades that protein folding is but a brick in the whole process.
In particular the statement " It is very, very rare for knowledge of a protein’s structure to be any sort of rate-limiting step in a drug discovery project!" does not reflect the realities of drug discovery.
Knowing a protein's structure, and its structure when complexed with ligands/drugs, is a massively important bit of data in the armoury of medicinal chemists, which Derek Lowe knows all too well.
Of course, it may be that knowing which protein to target is more important, and that problem isn't affected by AlphaFold.
Are you working in the field, or is that a dilettante's opinion?