Hacker News Reader

Comment by nybsjytm | Hacker News Reader

Important caveat: it's only about 70% accurate. Why doesn't the press release say this explicitly? It seems intentionally misleading to only report accuracy relative to existing methods, which apparently are just not so good (30%, 50% in various settings). https://www.fastcompany.com/91120456/deepmind-alphafold-3-dn...

View on HN

They also had a headline for Alphazero that convinced everyone that they crushed Stockfish and that classical chess engines were stuff of the past, when in fact it was about 50 elo better than the Stockfish version they were testing against, or roughly the same as how much Stockfish improves each year.

And then what happened is AlphaZero changed the professional game in various interesting ways, and all its ideas were absorbed into Stockfish. A little bombast is forgivable for technology that goes on to have a big impact, and I don’t doubt it’s the same story here.

> all its ideas were absorbed into Stockfish

don't think that is true, Stockfish incorporated NNUE techniques through a fork https://www.chess.com/news/view/stockfishnnue-strongest-ches...

being transparent with the setup of your invention is always a good thing.

>all its ideas were absorbed into Stockfish

That's not true at all, Stockfish still uses only human heuristics for search and NNUE for eval, a completely different architecture than alphazero and derived from the Yu Nasu Shogi engine.

It's a neural network trained on self-play games (many of them lifted from Leela Zero). I get that it's a different shape of network, but people really seem touchy about crediting Google with the kick up the bum that led us here. AlphaZero had a massive effect on chess globally, whatever people think about its press releases. My main point is that people should update the heuristic that wastes energy arguing about bold claims when clearly something amazing has happened that everyone in the industry will react to and learn from.

I don't have any particular thoughts about DeepMind's board game algorithms or how they were advertised, but even if I happened to think it was the most innovative and influential research in years, I'd still ask for honest communication about the work. It's part of being a healthy research community - although clearly the AI community falls well short on this, and nobody could say it's only DeepMind's fault.

>It's a neural network trained on self-play games

It's not, it's just supervised learning on evaluations. There is no self-play involved when training the model.

Where do the evaluations come from? The idea that Stockfish isn't benefiting hugely from Google having created and advertised AlphaZero is preposterous, can we please just stop?

>Where do the evaluations come from?

Good datasets are selected empirically, they are usually a mix of different sources, not a single engine.

>The idea that Stockfish isn't benefiting hugely from Google having created and advertised AlphaZero is preposterous, can we please just stop?

I have not said anything about AlphaZero, I am just reporting where you are wrong. Your arguments are simply not very convincing.

Okay, well, no sale I guess. Stockfish's training dataset is mostly self-play games from an engine directly inspired by AlphaZero. It moved to neural network evaluation after a fork based on a paper that cites AlphaZero. It plays chess more like AlphaZero than Stockfish 11. Yes, it's extremely interesting that it continues to edge out Leela with a fast, rough approximation of the latter's evaluation but much faster search. But it (and human chess) wouldn't be where it is today without AlphaZero, and I was originally responding to someone dismissing it based on the perceived over-zealousness of its marketing, as people seem to want to do with TFA. I merely submit that both of these Google innovations are exciting and impactful, and we should forgive their presentation, which nevertheless has been kind enough to link to the original papers which have all the information we need to help change the world.

> someone dismissing it based on the perceived over-zealousness of its marketing, as people seem to want to do with TFA

Sorry, but that's nothing but a reading comprehension problem for you

A lot of that going around. Have a great weekend.

I think Alphazero is a lot more interesting than Stockfish though. Most notably it lead me to reevaluate positional play. Iirc A0 at around 2-3 ply is still above SuperGM Level which is pretty mind-blowing. Based on this I have increased my strategy to tactics ratio quite a bit. FWIW Stockfish is always evolving and adapting and has incorporated ideas from A0.

Stockfish has not incorporated ideas from AlphaZero. Stockfish's NN eval technology, NNUE, comes from Shogi and it predates Alphazero there.

The 2nd strongest engine, Leela Chess Zero, is indeed directly inspired by AlphaZero, though, and did surpass Stockfish until NNUE was introduced.

Hmm: NNUE was introduced in 2018, the AlphaZero preprint 2017, AlphaGo 2015-2016. I checked this because my memory claimed that it was AlphaGo's success that sparked the new level of interest in NN evaluation.

Wouldn't surprise me if AlphaZero's improvements had no influence in that timeline, but for AlphaGo it would.

The original NNUE paper cites AlphaZero[0]. The architectures are different because NNUE is optimized for CPUs and uses integer quantization and a much smaller network. I don't think one could credibly claim that it would have come about if not for Google making so much noise about their neural network efforts in Go, Chess and Shogi.

0: https://github.com/asdfjkl/nnue/blob/main/nnue_en.pdf

For whatever it's worth, the NNUE training dataset contains positions from Leela games and several generations of self-play. Stockfish wouldn't be where it is if not for Google's impact. AlphaFold will likely have a similar impact on our understanding of protein structure. I don't know why everyone is so offended by them puffing their chests out a little bit here, the paper's linked in the article.

How do you train for strategic thinking in chess? I read a book on positional chess once, but that's as far as I've gone.

The first thing I'd recommend is constantly evaluating positions from a strategic POV ("Evaluate like a GM" is a good book, alternatively look at a lot of positions and evaluate like you were an engine and then engine check).

Second (or first if you lack even the basics to do said evaluation) is understand strategic concepts. A good starting point would be "Simple Chess" the next step would be pawn structures ("Power of Pawns" -> "Chess Structures" would be my recommendations, the latter is probably the greatest chess book in recent times imo). There's also many Chessable courses, I'm quite fond of "Developing Chess Intuition" by GM Raven Sturt and the "Art of..." series by CM Can Kabadayi for lower rated players. The sky is the limit, there's good books all the way up, for example "Mastering Chess Strategy" usually recommended for 2000+ ELO

Third study great positional players like Carlsen, Karpov, Petrosian etc.

I'd say the most important thing to realize is that just like tactics puzzles, there's strategic puzzles but they are not as obvious.

Thanks.

Alphazero indeed crushed stockfish with a novel technique, I think it deserved all the praise.

It definitely deserved a lot of praise, but the testing situation wasn't really against a fully fledged stockfish running on similar hardware, but one that, among other things, had no opening library.

The issue is not whether alphazero was impressive, but that we should be careful about the specific claims of the press releases, as they are known to oversell. The whole thing would have been impressive enough if the games had been against the last release of stockfish with good hardware, just for the way it played.

That's what I thought. They go from "predicting all of life's molecules" to "it's a 50% improvement...and we HOPE to...transform drug discovery..."

Seems unfortunately typical of Google these days: "Gemini will destroy GPT-4..."

IIRC the next best models all have all been using AlphaFold 2's methodology, so that's still a massive improvement.

Edit: I see now that you're probably objecting to the headline that got edited on HN.

Not just the headline, the whole press release. And not questioning that it's a big improvement.

That's pretty good. Based on the previous performance improvements of Alpha-- models, it'll be nearing 100% in the next couple of years.

Just "Alpha-- models" in general?? That's not a remotely reasonable way to reason about it. Even if it were, why should it stop DeepMind from clearly communicating accuracy?

The way I think about this (specifically, deepmind not publishing their code or sharing their exact experimental results): advanced science is a game played by the most sophisticated actors in the world. Demis is one of those actors, and he plays the games those actors play better than anybody else I've ever seen. Those actors don't care much about the details of any specific system's accuracy: they care to know that it's possible to do this, and some general numbers about how well it works, and some hints what approaches they should take. And Nature, like other top journals, is more than willing to publish articles like this because they know it stimulates the most competitive players to bring their best games.

(I'm not defending this approach, just making an observation)

I think it's important to qualify that the relevant "game" is not advanced science per se; the game is business whose product is science. The aim isn't to do novel science; it's to do something which can be advertised as novel science. That isn't to cast aspersions on the personal motivations of Hassabis or any other individual researcher working there (which itself isn't to remove their responsibilities to public understanding); it's to cast aspersions on the structure that they're part of. And it's not to say that they can't produce novel or important science as part of their work there. And it's also not to say that the same tension isn't often present in the science world - but I think it's present to an extreme degree at DeepMind.

(Sometimes the distinction between novel science and advertisably novel science is very important, as seems to be the case in the "new materials" research dopylitty linked to in these comments: here https://www.404media.co/google-says-it-discovered-millions-o...)

Anyone remember how he marketed his computer games?

No, how?

Massively overpromising unachievable things. From wikipedia:

https://en.wikipedia.org/wiki/Republic:_The_Revolution

Initial previews of Republic in 2000 focused upon the purported level of detail behind the game's engine, the "Totality Engine". Described as "the most advanced graphics engine ever seen, (with) no upper bound of on the number of polygons and objects", it was claimed the game could "render scenes with an unlimited number of polygons in real time".[14] Tech demonstrations of Republic at this time showcased a high polygonal level of detail,[21] with the claim that players would be able to zoom smoothly from the buildings in Novistrana to assets such as flowers upon the balconies of buildings with no loss of detail.[22] The game was further purported to have artificial intelligence that would simulate "approximately one million individual citizens" at a high level of detail,[23][19] each with "their own unique and specific AI" comprising "their own daily routine, emotions, beliefs and loyalties"

I feel like it's always worth bearing in mind when he talks about upcoming capability.

Thanks, I'd never heard about this before. I definitely think it helps in understanding his commentary. It's really a shame that DeepMind picked up his communication style.

I'm quite hyped for the upcoming BetaFold, or even ReleaseCandidateFold models. They just have to be great.

> it'll be nearing 100% in the next couple of years.

What are you basing this on? There is no established "moores law" for computational models.

It's the internet. There is no source more cited than "trust me bro"

Computational models have been shown to improve with computing power though, right?

It's a tongue in cheek comment about how fast models have been improving over the last few years, but I forgot HN scrutinizes every comment like it's a scientific paper.

The ROI exponentially decreases with language models. At this point, each percentage point of accuracy costs you tens of billions, and the projections show a solid wall approaching with the current implementation tricks.

It's an off the cuff comment, at this point though, HN apparently needs to bully everyone who refuses to go along with the zeitgeist, as if being negative near a thing would destroy it.

I guess the 'h' no longer stands for 'hacker.'

Which specific AlphaX model evolved like that? Most of the ones that were in the press had essentially a single showing, typically very good, but didn't really improve after that.

Reply on news.ycombinator.com