Hacker News Reader

Comment by resiros | Hacker News Reader

"Now we finally get to something interesting: how AF2 actually works! Alas, I will be able to say a lot less than I had hoped for, and here I have to do something which I very much dislike to do but feel that I must—call out DeepMind for falling short of the standards of academic communication. What was presented on AF2 at CASP14 barely resembled a methods talk. It was exceedingly high-level, heavy on ideas and insinuations but almost entirely devoid of detail"

I agree 100% with this paragraph. I have watched alphafold presentation and was as disappointed as Mohammed. Deepmind kept all the interesting parts of their method secret and Jumper dodged question after the next about the method. I am disappointed because deepmind would not had any chance solving this problem without the CASP community and advances shared over more than 20 years.

View on HN

I fully agree with this as well. This is why I think it’s important for research to be carried out by academia: they have no incentive to hide their secret sauce.

Yep. This is soooo sad. The mighty power of Google allows it to, once again, be the winner takes it all. And unfortunately, get into a position where results could be protected. So Google will probably leverage his commercial advantage for some years before enough details surface so that one can rebuild a solution that can be shared freely.

Note this protein folding thing was not solved by other mighty giants such as Johnson&Johnson, Pfizer, etc.

So basically, Google has even more power than before.

But to be honest, energy production knowledge (nuclear stuff) is not in the hands of governments but in the military-industrial complex... So it's history repeating...

The civilizational struggle lens feels a bit overwrought, OP notes this was on the competition's timeline, not AlphaFold's, and Deep Mind is rushing a paper to follow soon.

There’s plenty of incentive if you are at risk of getting ‘scooped’.. especially if your livelihood depends on a publish or perish model.

I don't quite agree with this. I work in this field and it is quite rare to get the code along with paper by people from academia. In fact many a times big companies tries to replicate the the published work and publish the code/model like say TF Model Garden.

It depends on what you mean by "the field". Do you mean machine learning? That might be the case given the corporate dominance of that technique. But in bioinformatics (the domain to which AlphaFold2 is applying ML to) it is generally expected that all data and code be made available for a tool. Otherwise what's the point? If people can't build upon it, science hasn't been done.

You're overstating the situation. The previous top-performing protein folding method is essentially impossible to reproduce from public literature, and available only via licensing agreement with the lab that created it.

More generally, while it's far more common to find open-source implementations of lab research than it used to be, it's still quite common for code to be closed, and important methodological details to be secret.

Even in the "open" fields of ML, today, I routinely find papers where essential bits of code, parameter sets, etc. are not made available.

In bioinformatics and academic protein folding in particular, there has been a strong tradition of sharing work and building on others' work.

That's arguably why CASP exists, as part of a recognition that the field benefits from collaborative effort to solve the problem - e.g. via public identification of what approaches have worked.

How come the next best protein folding software is proprietary, then?

Lots of AI competitions have requirements that competitors are open source, see for example SAT Race ( http://sat-race-2019.ciirc.cvut.cz/index.php ). This seems to be less the norm in deep learning areas, which is unfortunate.

What is "in" and what is "out" of academia. Plenty of important research has been conducted in industry, for centuries, from the invention of student's t test through everything that irving langmuir discovered and nyquist sampling theorem. Some things have been discovered in "basically not academia", too, like the chemiosmotic effect.

I've seen many academics hide the secret sauce. Getting tenure is competitive and any paper you can publish that gets lots of cites is important. Leaving out tiny details to replicate work is common.

DeepMind should re-read their own policies: https://deepmind.com/about/ethics-and-society

Which part specifically? A quick read doesn't indicate anything about being open about the details of their methods. Mostly the policies are about the ethics of how AI is built and used.

> We start from the belief that AI should be used for socially beneficial purposes and always remain under meaningful human control. Understanding what this means in practice is essential.

> We embrace scientific values like transparency, freedom of thought, and the equality of access...

That's wildly different from making everything publicly available.

Not if you claim it is essential to "understand what it means in practice"?

But it is not. It is essential to do the same thing without spending any effort yourself, it's not essential for understanding the science.

Is it a scientifc result if it isn't reproducible? Would the result have been accepted if it was submitted by an unknown?

The reproducible part is that they can consistently fold proteins - which they can.

The fact that they're not telling anybody else how they did it stops it from being public science, but doesn't stop it from being science.

There are two terms that may be conflated in this particular discussion.

Reproducible: Can it be done again?

Replicable: Can someone else do it?

AF2 appears to be reproducible, but not yet replicable.

https://phys.org/news/2019-05-replicability-science.html#:~:....

> Can someone else do it? > but not yet replicable.

Why not? I don't see a blocker except for the money to create a team and the computational resources. I think AlphaGo/Zero was harder to replicate than this, still Leela's team managed to get it working.

They mentioned that they'd publish a paper soon. Why is it a problem, then?

Reply on news.ycombinator.com