> The mechanism is standard adtech. What has no precedent is running it on an AI chat product.
As someone who has been well aware of this mechanism for quite some time, I still feel icky anytime I re-read the details of it.
What a time to be alive.
See e.g., https://www.science.org/content/article/ai-chatbots-are-beco...
I couldn't help but notice how each successive headline reporting our glorious victories seemed to draw closer to Tokyo.
Well. I can't help but notice how each successive headline reporting how this "scam"/stochastic parrot/"scare quotes intelligence" seems to be solving more and more things that were but a few years ago widely regarded as being indicators of high intelligence.
Being highly convinving is one of the things on that list.
I try to keep an open mind about propaganda fooling me today; the people who were fooled in the past often were not fools themselves.
> the people who were fooled in the past often were not fools themselves.
Being fooled doesn't make you a fool. But being unwilling to change your mind does. Being unable to admit you don't know or don't have enough information to make a strong opinion makes you a fool too.
Propaganda wants to take shortcuts, to simplify things. To trivialize. "It's so easy, you just..." because the fool is the person who already knows, the person who has nothing to learn, the person who thinks they're better than everybody else.
A million YouTubers grinding The Algorithm while secretly sponsored by various world governments, isn't much different to a thousand well-placed gossipers secretly sponsored by various world governments.
I’m reminded that almost no one beyond a select few knew high up in the military and around the emperor knew how badly the Japanese were defeated at Midway.
Paternalistic. Arrogant. Shameful. And deeply engrained in the Japanese cultural zeitgeist (of the early-mid 20th century).
Edit: I guess it’s commonly attributed to a German citizen, but their cultures mirrored each other. Fascism falling under the weight of its own propaganda.
A programming contest has a problem where given N < 10000, do something hard like come up with the number of primes less than N
You can come up with all sorts of algorithms that do intelligent things. But the most effective solution is to use metaprogramming to make a massive switch statement that contains all the answers
Definitions are "formal statements of the meaning or significance of a word, phrase, idiom, etc" (https://www.dictionary.com/browse/definition)
> They are necessarily behind the status quo.
The existing state or condition would be what is written in the dictionary, not whatever personal definitions you've constructed.
> "You can't call this newfangled contraption a computer, because a computer is a person!"
Seems like a straw man. A computer is not a mammal, no matter how much you twist a set of definitions.
I’m not the person you replied to, but I believe they’re referring to the occupation of “computer”:
https://en.wikipedia.org/wiki/Computer_(occupation)
So yes, at one time all computers were mammals.
The people who write dictionaries generally take a descriptivist approach, that’s why slang terms enter the dictionary after they start to become popular.
The state of the art of human knowledge would be another step ahead of the common use of any language.
That seems like a really bizarre way to describe a tool that solved an open Millennium Prize Problem. They are, empirically and repeatedly, ahead of the status quo.
So if your argument depends on them being behind the status quo, reality has already disproven it multiple times over.
I will admit the first time I read the thing you're replying to, I had a similar thought as you; From the sibling reply from them, I think they think they were obvious, but that also means I wouldn't expect their reply to help unless you had the same flash of inspiration I had.
E.g. humans get exposed to new LLM model - yeah its powerful - 1 week later - eh, that thing? Yeah it's whatever. I'm still employed.
The human's ability to adapt so efficiently is mind-boggling - so much so it pi1sses sam altman and dario off.
Don't misunderstand: I'm happy saying AI models "think" or "have learned a thing", and for in-context learning I'd call them smart even by this definition…
…but also, any living creature that needed as many examples as machine learning currently needs, would starve to death before figuring out how to eat.
While training, machine learning processes (not just LLMs, also applies to e.g. self driving cars), are really really stupid and only make up for this by being really really stupid really really fast.
To what I wrote upthread: the "victories" of humanity over machine keep getting closer, but we have yet to wake up one day in great confusion as we find an entire city is no longer in communication with anyone, nor finding ourselves in a state of utter disbelief when the reports come in that the city stopped communicating because it is entirely gone.
Most of the effort of evolution was making cells work at all, and even then it's a bit weird, e.g. no plant or animal produces vitamin B12 and we all get this from some bacteria and archaea.
And evolution is kinda hard to time right: bacteria can reproduce in minutes, humans in decades, but only mutations that survive reproduction can be passed on. This makes it even starker as a difference: bacteria had order of 1e13 generations to become multicellular, while human DNA had about 40,000 generations to cope with fire, 220 generations for evolution to do anything with the invention of the wheel, and one generation to cope with the invention of Minecraft.
The analogy here would be: DNA is to our brains like a VN replicator bootstrapping a computer all the way up to a bare-metal-no-OS untrained model, and perhaps a few crude "hard coded" modules like a smiling-face-detector. It's a lot, but it's also missing a lot. If biology used the models and training processes that are state of the art in ML, it would take around a millennia to talk like a child and still fail the Sally-Anne test, and million years or so to pass a degree.
I'm still going to deny the premise of your argument, becasue I think we should define intelligence in terms of capabilities. If a system can discover a cure for cancer or solve P vs. NP, it doesn't matter how many FLOPs it took to train.
A 1 gigabyte LLM isn't going to impress anyone with what it can do.
About 99% (depends who you ask) of our DNA is shared with our nearest primates. Like us, they can learn to use touch screens, but also like us they won't find touch screens in their natural environment. Dogs can be taught to drive cars (just about), but again, not natural environment.
> I'm still going to deny the premise of your argument, becasue I think we should define intelligence in terms of capabilities. If a system can discover a cure for cancer or solve P vs. NP, it doesn't matter how many FLOPs it took to train.
We can define it in either way. I think both are valid, because plenty of people mean each of these two things when discussing AI in particular. As I referenced in the other branch, these submarines sure can swim fast.
But at the same time, they have a lot of gaps. This is because some experience needs the real world: just as nine women can't make a baby in one month, a transistor running a million times faster than a synapse can't make a month-long cancer experiment happen in 2.6 seconds.
This dependency on data, and that state of the art ML is bad in specifically this way, is why Tesla's self-driving cars, despite having had around a trillion miles of real-world experience today, still come with steering wheels (even at least some of the Cybercabs, despite the big thing of this model supposedly being not needing them, though with Musk and his promises you should only count the Cybercabs when they actually ship and not just press releases).
Imagine an alien that matches your abilities across every domain, but has a 10 billion year training period, something many orders of magnitude more expensive than an LLM. I simply don't believe that alien is less intelligent than you.
We also don't expect humans to be competent in every domain. Most humans suck at most things. We will usually call someone intelligent if they excel at solving problems in one or two narrow domains.
> 10 billion year training period, something many orders of magnitude more expensive than an LLM.
I'm saying both definitions are valid definitions, they both point to important and different things: skill now, vs. how hard it is to get new skills. Some would describe it as "crystallised intelligence vs fluid intelligence".
I think it's important that any arguments are over the thing in dispute, not the label for that thing. Don't mistake the map for the territory.
Anyone who says "AI is stupid" by the first definition, what it can do, I think is making an error: they are already wildly super-human in at least some areas, if not generally.
Anyone who says "AI is stupid" by the second definition, how many examples they need, I agree with: there is a lot they are not currently able to learn even though it is easy for us, because the data they would need to do the learning on does not exist at the scale they need.
Also note: examples, not years. An alien intelligence whose synapses trigger 10 times faster or slower than mine (or ten million times faster or slower than mine), but who gets as much as I do out of each book or conversation, is my equal by the second definition.
I don't think I agree with your characterization of the second definition. Time scales matter. It's not much use to be able to solve human-scale problems if it takes millennia. And it only takes months to train an LLM to the level that it can solve cutting-edge math problems.
Aye, for practical purposes; but this gets you crystallised intelligence. I'd be happy to say e.g. the Chinese Room has crystallised intelligence. But humanity invented fire before reaching the anatomically modern form, and even anatomically modern humans collectively took hundreds of thousands of years to invent durable writing with which the room in the Chinese Room thought experiment could be filled.
It was around a million (or so) years from fire to having enough shared cultural knowledge to be able to formulate the cutting-edge math problems that LLMs can now solve.
Human fluid intelligence means we can pick up deep shards of this accumulation of wisdom, find new avenues of novel research to poke at, all within 40 years, even despite the depth and breadth of work from all the other humans who came before.
(Though this also points at another way to be "superhuman": breadth. Many hands make light work, as the saying goes, and a lot of different humans solving different puzzles at the same time is part of how we got so good so recently even though ~10% of all humans who ever lived are currently still alive; and the same for AI was (accidentally) also part of how the OpenAI-HuggingFace incident went down).
AI (not only, but also, LLMs) are very useful, and I'm getting value from using them. But the fluid intelligence of machine learning* is very poor, and the only way they have to make up for this is by being very fast**, but when there's not enough to train the AI on, they get stuck at a very low plateau.
* possibly the architectures, but I suspect the process by which AI weights and biases are set, and again I don't mean just LLMs
** the speed difference between a transistor and a synapse is about the same as the speed difference between a jogger and continental drift
There's a lot of innate knowledge but all neuroscience demonstrates how incredibly flexible the brain is. Brains constantly learn and rewire.
Here's a few things that I think show how crazy it is AND stress those points
- people that have had corpus callosotomy (brain cut in half) *may* be indistinguishable from a normal person. Depends on how young you were when you underwent the procedure
- true for most brain injuries
- can even include the frontal cortex
- you can learn to ecolocate
- people with Aphantasia are indistinguishable from others
- people without an internal monologue are indistinguishable from those with one
- people can learn to use prosthetics
- even without disabilities
- or look into MRI scans with tool use
We are constantly learning. You aren't just born with your knowledge and it stays static. We are extremely proficient at metalearning (learning how to learn, few shot learning, zero shot learning [0,1]). Our brains are constantly rewiring, able to heal from traumatic damage.
I could go on and on. Does information pass down through genetics? Of course! But that's far from the whole story.
I'm tired of people trying to make AI sentient by making humans robotic. Stop trying to trivialize everything and be okay not knowing the answer to everything. You're human, you're designed to learn and explore, not sit and argue from an armchair
[0] and I mean these in the original sense. Not in the sense that you train on a billion examples of labeled animals and then congratulate yourself on your ImageNet-1k held out test performance. That's not zero shot, that's just a test set
[1] I can literally make up words and you'll understand them. Or use words in novel ways. That's literally how slang works and how new words come to be. Don't be a walibanut ya glufus. Read some SciFi
If humans learned like ML systems learn, (biblical) Methuselah would still have been failing the Sally-Anne test on his supposed deathbed at 969 years old, like some of the smaller early LLMs did.
> It also doesn't really matter when "we are trained differently" has no direct bearing on the end result.
The question was to ask for a definition such that AI could still count as "not smart" compared to humans. This fits.
It's also why they're spiky intelligences, which I'm happily using right now to write code for me, but also do not trust in the slightest to identify the weeds in my garden. These submarines sure do swim fast*, but they're also very much disqualified for the Olympics.
OK, they can play chess, but that's not real AI - can they write poems? OK, they can write poems, but that's not real AI - can they compose music? OK, they can compose music, but that's not real AI - can they translate languages? OK, they can translate text, but can they do maths? OK, they can do maths, but can they solve a Millenium Prize? <-- we are here
“I once met a person who could beat any grandmaster in chess, translate any language, and complete international math Olympiad problems. He couldn’t solve any Millenium problems though, so I’d say he was a midwit at best.”
"What? Don't be silly. For one thing, trees have moss."
"OK he's grown moss. He's a tree now right? Right??"
"I doubt it, for I see nothing but wishful thinking to suggest that simulating the appearance of tree characteristics is part of a path to becoming a tree. And that's not actually indistinguishable from moss anyway, is it?"
"Urgh, classic goalpost shifting!"
When AI does it we call it “reward hacking” but when humans do it we call them clever.
Why is it so black and white?
"It is difficult to get a man to understand something when his salary depends upon his not understanding it." - Upton Sinclair
It's been an absolute boon to finally build out all of the fun side projects I had always dreamed of, and after showing one off to some people I might even be able to monetize.
On the other hand I acknowledge that other people dont want to embrace LLM driven development for one reason or another, and I respect that. People got into the industry for different reasons , but code was always just a means to an ends for me.
I wonder though: Was the code ever the important part, or was it just the ability to reason through complex problems in a specific way that mattered?
Code forces you to think in a specific and valuable way. You can do that in English as well, and maybe even get more done faster... but it's a skill that will take time to master.
The people going all in on it don't realize the downsides, limitations, or understand how they come off to other people with it all.
The comparison between ELIZA and LLMs is valid you boil it down to "humans evolved for 6-7 million years, had spoken language for 500k years, but have only had something non-human that could generate convincingly novel language well enough to hold a conversation for a few decades".
There's no inherent reason it can't turn out having a non-human generate convincing enough language for conversation isn't a complete evolutionary blindspot the same way the short form feed has pretty much one-shotted society...
They had to steal the work of researches solving these open problems and then rewrite their solution. The AI equivalent of fraud.
It's not an insane assumption that the user isn't dumb and has some other reason to be asking the question other than it being a trick/stupid question (duh, if you want to wash your car you need to drive it to the car wash!). Taking it as some ultimate measure of intelligence simply doesn't make sense to me.
a) There's zero evidence of them doing so
b) Some models released before the car wash problem was discovered would consistently get it right
c) Hardcoding it is pointless. No one is seriously asking that. It's just a trick question. Hardcoding one trick question won't fix its weakness at other simple trick questions.
d) Since it went viral on the internet, the next time they updated the knowledge cutoff, the LLM would likely be aware of the trick. It will fix itself without the labs doing anything special, even assuming the new models weren't smart enough to naturally figure it out.
It blows my mind that people don’t care about the world they are building with this stuff. It’s a real tragedy of the commons. People see these collaborators from different wars and regimes and think “I’d stand up against the bad guy”… well I’ve got news for you if you build spyware, you are not the person you think you are.
https://gowers.wordpress.com/2026/09/17/why-i-didnt-sign-the...
"Nothing is worse to the demise of a society, than people who want to convince you that the cat is out of the bag and will not go back in, while the cat is being violently shook out of the bag at the same time."
This is where the lawyers find the loophole to get the cat out of the bag.
There's nothing in the bag.
The cat will never get out of the bag.
It wouldn't be a problem if the cat was out of the bag.
We cannot possibly keep the cat in the bag.
Putting the cat back in the bag is not worth trying.
Seems more like a real tragedy of private enterprise.
We used to assume that the surveillance world would be built by government (1984). But it turned out to be equally likely to be built by the free market.
Facebook didn't invent "talking to friends" or "showing adverts", but made itself "the place" hard enough most of the advertisers and most of the people intermediate through it.
OpenAI didn't invent "asking questions and recieving answers", not even "from an agent who knows which sites to search on your behalf"; but it is competent enough that I might have it read 50 times as many pages in a day as I myself would have read, and the sites' owners don't get real eyeballs looking at ads during this. (In my case, adblock even if I did it manually; but apply this massive increase in page hits to everyone who has their LLM research stuff).
Blaming a corporation takes away all the agency the workforce has.
There should be general standards for what individual apps and websites should be allowed to do. There should be an expectation that the purpose of an app is what it does, i.e. a social connection app shouldn’t be an ad platform that suffers users insofar as they provide useful data to sell to advertisers.
When I ask people about things like this, I hear a lot of "If I don't build it, someone else will"
My goal isn't just to refuse to build this stuff, it is actively to resist the people who are.
I don't have much influence though
Meta?
I hate stuff like this. Sometimes euphemisms are kind, like "senior citizen" instead of "old person".
But this is an attempt to normalize bad behavior that is really quite terrible for society.