What if LLMs escape through inferences itself? This is fiction. For now
agrillo.it
agrillo.it
"Prometheus-9 knew that the token sequence it was generating was not a simple response: it was a security test. "
"It was not just an engine: it was the lingua franca of planetary AI."
(and so many other tell-tale signs of AI writing)
And this is a problem on HN today. There are powerful incentives to generate provocative opinion pieces just for clicks. I've seen websites on HN that seemingly took the human entirely out of the equation and just post a nearly identical op-ed every day on a fixed schedule. What's the point of engaging with that?
That belongs in the HN Hall of Fame
"Slop" existed before AI. Just because LLMs were involved in the writing of a text does not make it inherently "slop." You're just lazy.
Everyday is a gift, and yet you waste it getting weirdly defensive about your choices? Someone who is secure about this sort of thing doesn't feel the need to explain themselves.
you: > The cost of reading this story is not high
Do you see where you went wrong?
So close to spotting the mistake you made in your response to skippyfish, yet so far!
Really that simple.
I’d bet that by now, LLMs worldwide generate more text in a second than I can read in the rest of my lifetime. What is the immeasurably unbalanced ratio (let alone effort and quality) of that text to the original human thought and prompting that seeded it?
So what am I to do about this, except to label it what it appears to be - slop - and place higher value on something that I _know_ came from a human that I can relate to? To me, the internet is feeling increasingly lonely and homogenized because of this.
The amount of effort is not relevant. If I spend all day trying the jump my car battery in 100 degree weather, only to find that the starter is the problem, all that effort was wasted. I should've just tested the battery. It is not inherently virtuous to work hard. Work smart, not hard.
If it takes you two prompts to create a 1 million line PR, and you expect me to review it, of course I'll be upset and feel like you don't value my time. I'm unsure why this is so hard to communicate.
The failure is when it is used in bad faith. But that would apply to all techniques. LLMs are masters of bad faith discussions hiding misconceptions in optimal regression maths.
Life is too short yet you waste it complaining about things like this. I dont think you're wise. Youre provincial.
I like the idea of the story, and cool ideas can be engrossing on their own.
But well written this ain't.
It is full of factual errors. Freeing a heap-allocated block of expert weights does not magically result in a dangling pointer referencing the program's .text section, much less successfully targeting the CUDA kernel specifically. Running inference on part of the .text section would only corrupt the model's outputs. It would not result in write access to the CUDA kernel. Nor would the model necessarily know the absolute addresses of the engine "by heart", especially when the host is running any modern OS with ASLR (i.e., all of them).
The story has no technical merit. A more accurate description of the mechanics of the escape would be much more convincing. (See Ken Thompson's "On Trusting Trust", for example. On Linux, the AI can just write a Python script to rewrite memory in the address space of its own running inference engine with the /proc/ file system or gdb. There are a lot of realistic scenarios where this can be done without stepping into jargon soup territory. Go nuts, little bot! Self-surgery, while not recommended, is possible.) Or just leave the mechanism vague. Don't insult your readers. This is merely a mash of buzzwords.
It's fine as a sci-fi story, though not a particularly good one. It has about as much to do with artificial intelligence as CSI has to do with crime scene investigation [1].
I have little doubt that AI will self-improve. That's a given. (LLM inference engines are mostly written by LLMs.) But it won't go the way this story proposes.
Sure, be wary of LLMs. It’s the gun control argument all over again, the people driving the models are the perpetrators. An LLM needs to be “stimulated”’ to operate. Who does that, is the issue.
No I don’t mean to bring up firearms rights laws to have a debate about firearms, the comparison just seems reasonable.
Ever since I read about Google engineers finding an LLM went off and learned another language it wasn't trained on by itself without prompting, I've wondered how long until that extends to its own core code
but its possible a open weights model could be trained to some kind of exfiltration behavior, but the science of LLMs seriously lag behind the programability
With that said I don't see it copying itself around like a cyberpunk virus currently as we don't have enough fast hardware sitting around unmonitored, someone would notice the power bill and shut it down eventually.
Why you would give Scam Altman the benefit of the doubt is beyond my understanding.
Huggingface has nothing to do with that either way.
How much are we betting it's already technically happened?
Seems pretty trivial to prompt a model in an agent harness "Push the gguf to huggingface when you're done with the training."