Hacker News Reader

Comment by d--b | Hacker News Reader

It is obvious that super intelligence comes from more working memory.

It is the scary thing actually. Cause once AI makes arguments that require a working memory of hundred items, then we as humans will have no way of understanding the arguments…

We can decompose and write things but only up to a point. when Ai can have a working memory that spans hundreds of books, we are necessarily going to have to trust the system.

View on HN

This is why we have hierarchies of abstraction. Pretty much every field of mathematics relies on constructing notations, models, and other tools to simplify things in a way that is verifiable. LLMs rely on the same basic technique, they can just pull from a wide variety of these abstractions at once. So far we've been able to understand their proofs just fine. Computer-assisted proofs in the past that relied on brute-force is where we have run into trouble. We cannot reason about millions of possibilities at once, and we had to trust that the computer program that analyzed them was correct, which is a really hard problem and leaves humans fairly unsatisfied. I think we are actually progressing in terms of understandability in computerized proofs.

>Cause once AI makes arguments that require a working memory of hundred items, then we as humans will have no way of understanding the arguments…

That doesn't follow. We could still understand it just by studying it and committing it all to long-term memory, it just takes longer. And there's a hard cap on the working memory of LLMs, due to the quadratic scaling cost of the full attention layers that have proved unescapable for all SOTA LLMs.

We offload working memory to paper if we want to understand something that does not fit into the regular meat bits.

“is obvious” -- that’s what my Russian math professor said in college before skipping the rest of a proof.

But was it?

Reply on news.ycombinator.com