Hacker News Reader

Comment by woeirua | Hacker News Reader

This is a big step forward, but the outstanding question as far as to whether or not this is useful for evaluating novel proteins, is going to be how good is the confidence metric at telling the user to trust or not trust the results. You can see from their examples, that AlphaFold is very good but not perfect. I imagine for some proteins it will still give misleading or erroneous results and if you can’t tell when that happens without verifying the structure experimentally then this will likely not be that useful for new science.

View on HN

> the outstanding question as far as to whether or not this is useful for evaluating novel proteins

That is not an outstanding question. The test on which DeepMind scored high marks is a test of how well the algorithm folds novel proteins -- proteins whose ground-truth structure has not yet been published.

We’d have to see the distribution of GDT scores evaluated on unknown proteins to say anything about how confident we can be. If the distribution is tightly distributed around the median then great, this works really well. If the variance is large though then you’re going to have a hard time using this for meaningful predictions.

According to the article there's a confidence score as well. As long as this is sufficiently predictive of errors either a tight or wide distribution is likely acceptable.

We need to see the relationship between confidence and GDT score. If you have a nice relationship then again everything is great. But... most confidence metrics from neural networks do not have a nice relationship to the primary metric.

You missed the actual outstanding question in their comment:

> the outstanding question ... is going to be how good is the confidence metric at telling the user to trust or not trust the results.

You don't generally look at neural network output like that.

There is generally a threshold, less than X, not the class, equal or more, is the class. Then you run the network with the same threshold on a known data set and compute a confusion matrix, which tells you about the error, I don't even want to know what a confusion matrix analogue for 3D geometry would look like but I'm sure they have something.

This is literally the process that one does in taking part of the this. And the error rate (specifically the lack of errors) is what is everybody is talking about. 90 is just as accurate as we can get with experimental measurement. It's likely at this point the source of error is in the data set (we can only train on data we experimentally measure and these are not perfect measurements). It's also possible, at this point, the model generalized so well that when it deviates from experimental measurements it's actually correct and the experimental value was the one that was wrong.

So no, the outstanding question is not "is going to be how good is the confidence metric at telling the user to trust or not trust the results.". Nobody is going to be looking confidence values when it model is giving an output, they are going to be looking at the overall error rate across a broad spectrum of proteins to get a sense of it's accuracy.

Scientists can verify that an AlphaFold-predicted structure is correct, or at least useful, without being able to get the structure experimentally. For instance, we could use the AlphaFold-predicted structure to do protein-ligand binding calculations for a bunch of known molecules. If these calculations agree with experimental protein-ligand binding (which they generally do for proteins with known structures), then we can say with high confidence that we've got a good structure.

does that mean that protein-folding is sort of in NP?

The way computer scientists do it, yes, it is. In the CS situation you define an energy function (in this case representing the physical behavior of the protein in water) and find a heuristic to approximate the coordinates of the lowest energy configuration; done, problem solved.

in reality, that's not how it works at all. The energy functions we have are crappy and require too much sampling before we can find the lowest energy configuration. And more importantly, it doesn't look like proteins typically fold to their lowest energy configuration (with the exception of some small fast two state folders), but rather explore a kinetically accessible region around there (or even somewhere else entirely, if the energy cost to transition is too high).

Methods like AF depend heavily on large amount of information correlation from evolutionary data, which has historically been of the highest value for making decisions about protein structure.

It’s probably not in NP, in that there is not a polynomial time algorithm that checks solutions for correctness.

I was wondering the same thing. But I also wonder if having good guesses makes the x-ray crystallography and other experiments to verify a given protein easier/cheaper/quicker? I don't know enough about the actual techniques to have an informed opinion but I would think it would be helpful.

It does. https://www.nature.com/articles/d41586-020-03348-4 reports a case of x-ray crystallography helped by AlphaFold prediction.

Every simulator is going to have error. In this case this biennial challenge represents the computational state of the art with scores of 30-40 over the last decade. The AlphaFold2 model sends that score up to 87 with errors about than the width of the atom. You can actually see the difference between their prediction and the actual result and it’s stunning. This is all on the blog site so I recommend reading before throwing shade.

I read the blog. But there’s a big difference between a mind blowing tech demo and something that can be used in a commercially viable process.

There's a difference between being a random commentator on HN and being one of the several experts in the field quoted in the article, among other things, predicting a mass exodus from the computational biology field as the major problem of that field is now solved.

Are you always so dismissive of Nobel-level achievements?

It’s a good question, and I’m not a domain expert here.

The article did claim:

> According to Professor Moult, a score of around 90 GDT is informally considered to be competitive with results obtained from experimental methods.

So perhaps their score of 87 GDT is pretty significant. But “competitive with” is not the same as “always in agreement with”, as you point out. Could be the failure modes are problematic.

There are other experimental methods that are much cheaper that can be used to assist validation. Also the models look damn impressive, even down to the sidechain packing.

Reply on news.ycombinator.com