ask HN: How does an LLM like GPT3 perform addition/division?

I thought it apt to describe GPT3 as an alpha, because it did very well on language tasks and not so much on typical beta stuff, like math. Once it acquires well integrated hardcore logical solving abilities, then it'll not only be alpha or beta, but looking for ways to control its superhuman powers. Otherwise it'll go of on a tangent and kill itself in an infinite loop or put humanity in one.

On your question: how do we do it? We also have memorized the basics and apply a heuristic to do something more complicated, if necessary by analogy. Which is what GPT seems to be good at.

at present, my understanding is that it is purely linguistic. If you ask "what do you call the wife of a king", it will answer "queen" because that is the highest likelyhood next token. If you ask "what is 2 + 3", it will answer "5" because that is the highest likelyhood next token purely from a linguistic point of view, i.e. no math has been performed.

All the emergent properties of chatgpt being so good, is that it is very good at predicting word sequences that, when evaluated by a human, sound like it is doing all sorts of higher level logic.

Is uses the python shell. I asked it.

>what is 2.87863853+2.345435345?

>The sum of 2.87863853 and 2.345435345 is 5.224073875.

I can't find either of those sequences on the web, but it seems to able to add them together so it seems to have some logic ability, or they check for math terms and do math on it.