The fascinating this is that the LLM is not acting as a tool here AFAIk, but very much like a colleague.
I have no knowledge of the domain and have only PhD EE level math knowledge, so maybe my bar is too low.
What I do notice however is that LLMs are becoming capable of doing an increasing part of the intellectual work I can do, and usually a lot faster.
Just today I presented an agent framework that can take an informal incident statement and propose infrastructure changes to fix it, all evidence backed. This did nothing I could not to, but it did all 5 test cases in 6 - 12 minutes each. I would have found all of the monitoring indications it did, but it would have taken me a day per test case. The LLM also included sass to silly tickets. ("This is not even worth spending monitoring resources on. It's obviously a configuration problem.")
That's how this is reading to me as well. It's just fast at slogging through a certain level of "simple" transformations.
There is clearly intelligence there. We have no way to recognise intelligence other than the appearance of intelligence and this very clearly displays that.
It's also quite clearly different to human intelligence in some notable ways, but not in any that preclude describing it as intelligent. At least for normal non-pedantic definitions of the word.
But there's no way the thinking times would have been that short, of course.
that's some Harry Potter kind of "writes itself" book.
at this point, for me, any comment about LLMs that begins with "it's just ..." is hard to take seriously.
Terrence is impressed. Good enough for me.
I'm sure people thought calculators and, indeed, computers themselves were very Harry Potter as well when they first came out. But in the fullness of time the magic and mystique has drained away, and we're left with the understanding that they're just tools.
> at this point, for me, any comment about LLMs that begins with "it's just ..." is hard to take seriously.
Similarly I have a hard time taking seriously the people who make breathless claims of intelligence where there's only a text calculator with weights applied. It's like watching the devout cry "miracle!" at every strangely shaped piece of toast.
Terrence is impressed, but he's not a believer.
nobody thought that
A second corollary is that rational consciousness and thought is less likely to be contained in language than previously thought, because if language is so simple that a machine can process it, it can't contain consciousness.
Da hole raisin y nat-lang be v. hard is dat i kan rite lik dis an it be cool 4 native engrish speekrs 2 unerstand. LLMs are of course fine with this sentence in exactly the way that Zork's engine couldn't be.
example For, semi-randomise I word order can this like, Yoda worse than, and be understood.
> is no problem for a machine that takes context and probability into account when translating words to the underlying grammar structure.
We had to invent Transformers to be able to do that with reliability anything close to being worth caring about. Transformers have to learn from examples, not be pre-programmed.
It's clearly much more than that.
[0] https://gwern.net/scaling-hypothesis#gwern-difference--effic...
Exactly. The "most likely next" series of tokens, for example, when given the first half of a correct mathematical proof, is the correct rest of the proof. I have never seen anyone define "most likely next token" in such a way that this isn't true.
that feels like a misunderstanding of how the loss function behaves when used within a sequence
Either humans are not capable of intelligence or computers are capable of becoming intelligent. Neither or both.