AlphaFold at CASP13: What just happened? | Hacker News Reader

AlphaFold at CASP13: What just happened?

moalquraishi.wordpress.com

188 pointssytelus61 comments

>What is worse than academic groups getting scooped by DeepMind? The fact that the collective powers of Novartis, Merck, Pfizer, etc, with their hundreds of thousands (~million?) of employees, let an industrial lab that is a complete outsider to the field, with virtually no prior molecular sciences experience, come in and thoroughly beat them on a problem that is, quite frankly, of far greater importance to pharmaceuticals than it is to Alphabet. It is an indictment of the laughable “basic research” groups of these companies, which pay lip service to fundamental science but focus myopically on target-driven research that they managed to so badly embarrass themselves in this episode.

I wonder, is this because these methods are simply 'not good enough' to really have an application for medicine yet? I know nothing of the pharmaceutical sector, but saying they don't do basic research seems to stretch my world view given their vast profit baselines and government funding for exactly that purpose. Is there someone in the field who knows more?

The crucial observation is how their value chain works. Most of the value is in restricting the distribution of treatments. That, in turn, is achieved by getting medicines approved in the US.

There’s a lot of value near the end of the development cycle, and not near the start.

(I conclude from this that there remains a role for government in directly funding scientific research.)

I don't know why the author is so critical of his peers. DeepMind didn't come up with a novel biological insight; they simply pointed their unparalleled AI resources towards a deep learning problem. Is it really surprising that a team of world class deep learning scientists with virtually infinite resources managed to outperform pharmaceutical companies at a deep learning problem? I don't think so.

I think the key to the success was translating folding into deep learning problem and solving it (maybe there is still something to win by optimizing the models). The academia and industry were not treating folding as deep learning problem and they were using 'old' methods with slight improvements from year to year. The usage of deep learning was a breakthrough and will speed up research in this field.

That's most definitely not the case: https://scholar.google.com/scholar?q=deep+learning+protein+f...

The Go community was rather forthcoming to what AlphaGo brought, because it provided new insights to a field that some of them study quasi-religiously.

Let's see how pharmaceutical companies are dealing with that, or the next fields DeepMind will enter and shake up like that.

Eh... no? If you look at abstracts, Zhang group (the second place) also used "deep-learning based contact predictor". Really, the usage of deep learning was kind of standard long before CASP13.

If you read OP, AlphaFold's main innovation is predicting distance instead of contact (regression instead of binary classification), which was also independently developed by Xu, and further, using probability distribution over distance instead of simply choosing the best distance.

Yes, you wonder how Alphafold compares with the combined wisdom of Foldit. You start out with a secondary structure prediction from Zhang lab and ask a panel of experienced humans to fold up the protein into a plausible tertiary structure. As a human you have a good idea what larger pieces fit where, but AI still has problems with the large picture, cf. the leopard sofa problem.

While they might virtually have infinite resources, I would be quite interested in how much this actually cost. Can anyone share estimates?

It's not like pharmaceutical companies are strapped for cash.

If they don't invest in R+D it's a choice.

Why should they invest in r&d when someone like alphabet will do it for them?

I believe they do spend a lot of money on R&D. Someone came along with a new idea and applied it to their field.

Now that Alphabet has sown them a better way, money should pour into the new technique.

While big pharma does do internal r&d, de facto, new drugs and drug strategies largely come by letting the government fund researchers (typically professors at large research institutes), encouraging these professors to do a spin-off (I deliberately don't want to use "startup") to absorb the scientific risk, followed by an acquisition for IP.

Sure, but I'd be surprised to see a pharmaceutical company profitably competing for talent with FAANG in the race of hiring top ML scientists.

> DeepMind didn't come up with a novel biological insight; they simply pointed their unparalleled AI resources towards a deep learning problem.

What this essentially comes down to is a bunch of teams with high domain expertise and high technical capability was beaten by a team with higher comparative technical capability and lower comparative domain expertise. One has to wonder if one of the teams with higher domain expertise (ie work in the field every day) could achieve better results by improving their technical capabilities, or more aggressively applying their domain knowledge.

The question really comes down to whether DeepMind can beat any hard "game" (chess, go, protein folding etc) better than humans with deep domain expertise.

> What this essentially comes down to is a bunch of teams with high domain expertise and high technical capability was beaten by a team with higher comparative technical capability and lower comparative domain expertise.

The fact that the experts are entrenched in departmental trench wars and crippled by bureaucratic crap on every step is probably a contributing factor. If they were free to advance actual research instead of contributing to some middle manager's manager's agenda, thinks might look different.

This doesn't affect chess and go however ...

Regarding domain expertise, I would guess that this is something you can catch up with within a year.

That's what people outside drug development think. Meanwhile, 70 % of all drugs fail at Stage 2 testing. That's right, experienced pharmacologists at established companies go forward with compounds they believe will succeed, and still in 2 out of 3 cases they are not better than placebo.

I'd hasten to add it's the same in software. We think the experts are right on the edge of performance, accuracy, and success while using all the right tools - meanwhile from the inside, it's a shambles in all but the highest stakes environments with the most qualified teams.

I think the parent is talking about expertise for protein structure prediction, not drug development

I think you have a point. OP mentions "target-driven research". If you have a fixed target, you can do crystallography and get the structure directly, you don't need structure prediction. That is, I think pharma's core interest is closer to particular structures, not a general method to predict structures.

Elucidation a given protein’s structure is still a fraught process. Maybe the technology has improved, but when I was in that fiel 5 years ago, the going rate was “One PhD” per structure, i. e. 2 or t3 years of tinkering, with no guarantee of success.

By referring to structures per PhD, it is clear you are influenced by academia, where that is more true. But academia is more concerned with solving novel structures than industry.

In industry, often we have structures of related proteins, giving a couple advantages:

Template-based modeling is feasible, a category that AlphaFold didn't win because they didn't use templates; using templates gives better structure prediction than what their free modeling can do.

Crystallography conditions are often similar between related proteins and techniques such as molecular replacement make it so we can solve the phasing problem easily, which is often a roadblock as well in academia.

Surely cryo-em has helped a ton with that?

For the general question of pharma investing in structure prediction, I think participants in CASP overestimate the importance of structure. It is nice to have and there certainly are structure-driven projects, but docking is so poor that often computational models of how a molecule binds, even when you have a structure of a protein, are unreliable and there are plenty of case studies of them sending teams in the wrong direction. This would only be worse in the case of AlphaFold since, as the post shows, GDT_HA is still quite poor.

From my experience in research, pharma has found that cellular models and phenotypic assays are far more meaningful for pushing projects forward. So, there is far more interest in applying machine learning to that data than for building protein structures. And those same methods can be applied to target-based projects regardless of whether you have structure. And regardless of how flexible your protein is. Huge portions of structure-based modeling has no ability to deal with protein flexibility, even if you know there are open and closed conformations of the protein or a loop that adopts half a dozen configurations.

Basically, academics working on folding often believe far too much in the importance of structure in drug discovery. The author appears to fall into that category.

> academics working on folding often believe far too much in the importance of structure in drug discovery.

It reminds me of the 90's and early 2000's when many were convinced that sequencing the human genome would suddenly make drug discovery a simple problem.

The author does make a point of discussing the question of what business does a team like DeepMind have researching the folding problem? The solution is of no apparent value to the parent company Alphabet, and yet they were still funded. Perhaps this has to do with the attitudes or values of "modern" tech companies? Historically, there seems to have been a cyclically nature to the volume of basic research in industry, peaking with Bell Labs, sinking with the rise of Welch, and now coming back with the Googs and Facebooks.

You seem to be implying that the world would be better if they didn't do this research, that doing it as a tech company to just apply the tech is immoral. Is that what you really mean?

Well, one could put it cynically as Deep has developed a giant "cannon" that can be fired at different very hard but reasonably well defined problems. They've hit go, chess and similar things. Hitting proteins keeps their activity in the limelight. Essentially, DeepMind is demonstrating the value of the money Google paid to purchase it.

Cowsandmilks points out the limit of AlphaProtein above: https://news.ycombinator.com/item?id=18647187

The problem is one runs out of reasonably isolated problems with a reasonable metric.

The author appears to fall into that category.

They seem to acknowledge this:

The above graph is misleading in one way though because it is dependent on a specific metric, GDT_TS, which only measures gross topology. If we care about high resolution topology, which we certainly do for most practical applications, then a more appropriate metric is GDT_HA, and using it the picture looks a bit different:

[graph]

Still a good trendline, but much further down from a “solution”.

Another caveat is that both of these metrics measure global goodness of fit, which is important in terms of the basic scientific problem, but is often not indicative of functional utility. Local accuracy, for example the coordination of atoms in an active site or the localized change of conformation due to a mutation, is what is often sought when answering broader biological questions. Global metrics hide local discrepancy by diluting it in the sea of generally good agreement between experimental and predicted structures.

and

Even for MSA / family-level predictions, there is the question of desired accuracy, which hinges on the biological application. If one is predicting protein structures to ascertain their general fold for function classification, then high accuracy is unnecessary. If on the other hand the objective is to design small molecule drugs that bind proteins, which require ~1Å accuracy in the local pocket, it is unclear if we have made any detectable progress.

So, have you refuted the argument in anyway or just downplayed it?

I would have preferred that text in the comment you responded to had less of a trash talking feel to it.

Also it’s a bit overplayed because although these are special moments, there’s no shortage of examples in history where a new approach in one field is applied to get results previously unattainable in another.

However there’s no need for rationalizations. The result doesn’t generalize to anyone is smarter than anyone else, or that it’s not important because it’s not the super most important drug discovery tool.

It’s just a special moment where a technique is impacting another area of research. Which is part of a special period in history where AI/ML is evolving from being literally a joke that could defund your research, to finally having a broad set of useful applications.

We should be glad to see as many of these moments as possible from any discipline to any other.

You're right that the methods are simoly not good enough. Further a large fraction of pharma research has shifted to biologics which doesn't depend on structure prediction for new compounds, etc but they only need the hi res structure of their target protein (which more often than not is already done) to go on and generate the antibodies and make sure they direct the desired motif on the target.

I worked a few years in pharma R&D (Roche, largest R&D budget in this industry).

In a pharma setting, the 3D structure of a protein is mostly used to perform drug design (https://en.wikipedia.org/wiki/Drug_design#Computer-aided_dru...), i.e. trying to understand how a chemical will physically interact with a protein and thereby modify it's physiological function, in order to treat a disease.

The biggest problem comes from the fact that proteins are (1) non-static and very flexible and (2) don't exist in vacuum, they interact with a myriad of other entities in a living system. In other words, it's not because you know the structure of a protein and how to theoretically perturb it with a small molecule, that you have a drug. The large majority of structures predicted to be active against a protein target are not, when tested in a biological assay. The process helps, but ultimately it's a very empirical endeavor (test a ton of different chemicals in actual experiments, try to abstract some logic and move on from that). As a result, simply knowing the structure of a protein will not get you far down the line into finding a new drug.

On the resource topic: Even in a very large pharma setting, you will find only a dozen of scientists or so dedicated to the topic (out of tens of thousands employees), supporting many projects and with very little time to perform their own research. As a result, any team fully dedicated to the problem (like AlphaFold) can easily over-compete pharma. Most of the cost in drug discovery comes from dealing with patients and clinical trial. It's only at this stage that you'll know how your drug really works, and how it fits in the existing market and society (think of neuroscience for instance).

I don't want to undermine the protein structure field and AlphaFold results (it's fascinating), but pharma business model de facto relies very little on knowing a protein structure or not. It's also mostly useful to design small molecules, a class a bit out of fashion (biologics are the top-sellers in 2018, and new modalities are coming-up, like RNAs and gene editing for instance).

> biologics are the top-sellers in 2018

This is true, and big pharma will continue to invest very heavily here, since these medications contain living organisms and are much harder/more expensive to make into "generic" versions.

Biologics do not contain living organisms. For example monoclonal antibodies are considered biologics and are not alive, they can also easily be made “generic”.

AlphaFold is a prime example on the importance of cross-pollination between fields and the need to fund diversified research approaches, as well as inventing fundamentally new tools that are potentially applicable in a wide range of fields.

It appears that the bandwagon effect in science is real and unfortunately too prevalent. Conservatism is a powerful institutional force that directs prestige and importantly funding away from 'fringe' approaches. Fundamental innovation often stems from maverick thinkers who still need time and resources to prove their ideas' worth but too much resource (funding, time, talent, publication venues) tends to gravitate towards eking out 1% better performance from mainstream ideas (while important, industry will often fund this kind of research anyway).

In life sciences, SENS for example already took almost two decades just to start to establish itself and is still far from mainstream. All the while, 3-4 orders of magnitude more resources are expended to obtain those 1% improvements based on mainstream treatments with little chances of yielding significantly better health outcomes for the patient.

Decision science should compel us to invest more resources on risky projects with high upsides. Humanity can afford to risk investing 15-20% of research resources, if not more, to explore fundamentally new approaches.

https://en.m.wikipedia.org/wiki/Strategies_for_Engineered_Ne...

One has to admire the candor with which he talks about what must feel to him and many of his colleagues as an existential threat to his career and/or life's work. Impressive. In business (as in academia I guess) there is this constant nagging fear of being blindsided by a well-funded or brilliant competitor. When that happens I personally just want to get drunk or roll up into a ball or something.

>as an existential threat to his career and/or life's work.

Is it though? DeepMind stood on the shoulders of the giants: they made use of decades of biological research and wet lab experiments. There's much more to academic research than predictive data science, which is honestly not meant to be their expertise at all. It is exactly their research that enabled DeepMind to reduce the problem to a solvable deep learning problem, and it is still their research that can best leverage the results of the model. I think there's an important distinction between properly understanding and pursuing the science behind the problem and taking data and fitting an already formulated problem through a deep network.

While waiting for the paper, it seems to me "fitting an already formulated problem through a deep network" is not at all what AlphaFold did. Its main contribution is formulation of problem to be solved by a deep network.

What do you mean? Protein folding was already formulated as a deep learning problem in prior research. DeepMind used several engineering tricks that they previously used in their other DL work.

I am actually surprised at the way it was done, though. I would not have done it that way... I would have set up a 3 dimensional convolutional net and done training on time reversed melting simulation transitions.

> Second, regarding the question of how academic groups should respond scientifically to DeepMind’s entry, I suspect the right answer comes from evolution: adapt. Focus on problems that are less resource intensive, and that require key conceptual breakthroughs and less engineering.

I disagree with this notion a bit - the idea that academic researchers interested in managing their career should consider focusing on problems that are less resource intensive in the future to avoid having to compete with the resource advantage of a Deep Mind ...

I feel I've seen this position analogously expressed in various forms over time in the internet space "don't try to compete with $huge_tech_company on the internet because $scale challenges that they invested $megabucks into solving". This kind of statement was made over and over again as the internet explosion was going on. If you actually go back and look at the details you'll find stories like 'google built $inhouse technology so they could scale to 100 * x visits per day" for increasing values of x depending on the article publish date. But if you look again at that article in the context of a year or two after it was published, getting to x scale for the cited problem has become trivial with nearly off the shelf hardware and software designs.

It seems to me that computing resource advantage is something that should actually _nearly always diminish_ over time, especially when there is widely known/appreciated understanding of the value of the resource ...

Scientific research (in many fields) has been performance and design-test-efficiency bottlenecked for a long, long time -- the fact that there are now wider software trends able to support breaking past those bottlenecks in specific problems is not evidence to me that there is now a resource Emperor to whom all others must cow in fear, forever unable to compete head to head in related spaces ... if anything, the fact that more efficient computation approaches are viable _right now_ and known to be valuable, can spread into the academic groups as easily as the domain expertise of academia spread into Deep Mind's first foray int this problem space ...

Great insights from someone in the field. AlphaFold improvement is equivalent to combining past 2 CASP improvements. Ruminations on huge advantage that industrial labs have with top engineering talent and compute resources makes author wonder if its worth continue in academic lab.

“If I were to pick, I think about half of the performance improvement we see in AlphaFold comes from the simple ideas above, and about half from the sophisticated engineering of the distance-predicting neural network.”

“...with DeepMind’s entry I will have to reconsider, and from conversations with others this appears to be a nearly universal concern. Just like in machine learning, for some of us it will make sense to go into industrial labs, while for others it will mean staying in academia but shifting to entirely new problems or structure-proximal problems that avoid head-on competition with DeepMind.”

“...competitively-compensated research engineers with software and computer science expertise are almost entirely absent from academic labs, despite the critical role they play in industrial research labs. Much of AlphaFold’s success likely stems from the team’s ability to scale up model training to large systems, which in many ways is primarily a software engineering challenge. “

“For DeepMind’s group of ~10 researchers, with primarily (but certainly not exclusively) ML expertise, to so thoroughly route everyone surely demonstrates the structural inefficiency of academic science.”

I'm sure there's a good explanation for it, but calling a conference that occurs in 2018 "CASP13" seems like an unnecessary way to create some confusion.

Based on the sidebar here ( http://predictioncenter.org/index.cgi ), the explanation is apparently that CASP1 happened in 1994 and there has been one every two years since then.

I would say that pretty much any time a team comes in and improves CASP results over baseline it's a win. However, traditionally, it's been too hard for regulars to reproduce the results of the winning team- it's not simply reduced to a github repo you can run to generate new accurate structures, like some recent advances in image and drug discovery have been.

Papers are nice, but github code that runs is gold.

Another field is being upended by deep learning methods, that's what happened.

A small team from DeepMind with mainly AI/ML expertise won first place at a prominent academic competition, besting teams of experts by a surprising margin.

Academics who have invested a lifetime studying and working on the problem are suddenly wondering if their skills and experience are at risk of becoming less relevant. They're wondering, how could a bunch of neophytes pull this off? Is this just the opening salvo?

A natural reaction will be to dismiss this as "nothing new, just better engineering and clever hacking with more resources." Another reaction will be to dismiss deep learning techniques as "curve-fitting without insight."

Such dismissals are misguided in my view. Judging by how quickly deep learning methods have become the dominant approach for producing state-of-the-art results in other fields, I would expect the same thing to happen in this field.

Computers have ALWAYS been about solving problems. I have long told others that when going into computer science type fields that you should get a double major (or at least a minor) in some other field because you are most useful when you can take your computer knowledge and apply it to some completely different field.

I didn't take my own advice... My career has been about learning other fields to apply computers to it, and it has been a very interesting journey. Note fields is plural there, I have not stayed in the same industry (I think this means something but I don't know what)

I personally think that machine learning will have a broad effect on the legitimacy of the pharmaceutical industries and health care in general. Once real data about all of these drugs, treatments, and expensive procedures finally starts being collected and exposed, the public will see things as they are, a farce.

> real data about ... collected and exposed

Which is not a machine learning issue. Just collecting the information is a costly / time consuming issue which can't be outsourced to servers.

Right but the current innovation with regards to ml for medicine will force more data to be collected.

The amount of data you would require to do proper machine learning using current gradient descent techniques are orders of magnitude greater than that done in a drug study.

We have lots of ways to process and learn from data already. ML may make some things easier, but we're ready to learn more things right now with existing tools. Getting the data is the problem.

For drugs, we already do postmarketing surveillance. Whatever you think of pharmas, drugs are not a farce.

If the pharmaceutical industry were willing to fake efficacy results, surely they could do better than having over ½ their drugs fail Phase III efficacy trials. Especially since one thing everyone agrees on is that the best way to lower drug costs is to figure out which drugs are going to fail Phase III much, much earlier.

https://www.webmd.com/mental-health/news/20080227/antidepres...

"Automation is scary," says the very smart person, "for people who do menial labor. Their jobs will be replaced... I'm so glad I work in a STEM field that requires the kinds of thinking machines will never be able to do!"

Reply on news.ycombinator.com