Hacker News Reader

Comment by sswatson | Hacker News Reader

Good on you for spelling out this reasoning, but it is manifestly unsound. For a wide variety of values of X, people a few years ago had no reason to expect that LLMs would be capable of X. Yet here we are.

View on HN

In 1989, Gary Kasparov said that it was "ridiculous!" to suggest a computer would ever beat him at chess.

"Never shall I be beaten by a machine!”

In 1997 he lost to Deep Blue.

Yeah, and back then people moved the goal posts too, saying Deep Blue was just "brute-forcing" chess (which isn't even true since it's not a pure minimax search).

Deep Blue was brute forcing chess in the sense that AlphaGo wasn't brute forcing Go.

Both of them contained a search algorithm that explored some moves from each considered position, usually not all moves. Both of them contained logic (learned or programmed) to evaluate moves and/or positions.

The differences between them are many, but brute force doesn't enter into it in either case.

And today he's got salient observations on politics which hold much of his attention, and Deep Blue is shut off and has done nothing further.

Not a good argument for turning everything over to the Deep Blues. What's Deep Blue done for me lately?

This is something that could be demonstrated rather than just argued.

Train an LLM only on texts dated prior to Newton and see if it can create calculus, derrive the equations of motion, etc.

If you ask it about the nature of light and it directs you to do experiments with a prism I'd say we're really getting somewhere.

We tried this experiment with humans, back in the 17th century, and only a few[1] out of millions managed it given a whole human lifetime each.

[1] Obviously Newton counts as one. Leibniz like Newton figured out calculus. Other people did important work in dynamics though no one else's was as impressive as Newton's. But the vast majority of human-level intelligences trained on texts prior to Newton did not create calculus or derive the equations of motion or come close to doing either of those things.

Newton did it at 23 and there would have been very few people with mathematical training. The LLM would be trained on the entirety of recorded human knowledge and mathematics up to that point, and would get to use a lot more energy so it still has a massive material advantage over young Isaac. Yet I don't believe calculus would magically appear in its response.

A good way to look at it is to compare it to today: LLMs are already trained and are operationalizing a lot more mathematical knowledge than any human, including experts.

Why are they not coming up with paradigm shift in knowledge expression/discovery like humans did back then?

Are we just not prompting them right?

LLMs have been trained on a lot more data than any single human (text wise at least) for years now and these sort of results have only been possible for the latest crop of models in the past few months. Models get better as they get better.

The argument is whether models of today, suitably trained on pre-17th century data (if comparable quantity was available) would be able to "invent" calculus et cetera.

If we believe today's models are sufficiently capable to have been able to do so, why are we not getting these types of results today compared to the entire world knowledge and especially math?

Are research mathematicians simply not prompting LLMs in the right way?

Reply on news.ycombinator.com