> Interesting! Agents are the presumed bottleneck for recursive self-improvement.
They might be, but we haven't reached the local maximum yet in my opinion. Qwen 3.8 27b models are impressive despite their low parameter counts. The same scale curation in data and RL could produce substantial improvements in coding with models like Astra or Fable. I don't think we are there yet. I don't think even Sonnet 5.5 is there yet, despite being widely successful with agents and surpassing Opus 5.5 in some cases (disregarding that it is more expensive than Opus sometimes).
Agents do not improve the models, but they do improve coding capabilities. The obvious caveat is that labs would have to RL for everything to make the models more useful, as RL'ing for Javascript world doesn't seem to improve other fields. But still, it could be done, and it would have massive economical consequences.
> People keep implying this, but I've never seen a concrete past example of a product that was sold by scaring customers about it.
Scare tactics are treated as smoke, where the customers presume there is a fire. No one really believes that AI can kill them, yet by saying so OpenAI and Anthropic enjoyed possibly the biggest tech boom in history, despite how models could not even count the R's in strawberries at the time.
Similar story now; no one really believes that AI is going rogue and is about to destroy humanity, but scare tactics make people believe the capabilities are higher than they are.
> I see you doing serious mental gymnastics here. Consider the possibility that people invest because their numbers are good, and their numbers are good because their product is useful?? I mean, that is Occam's Razor.
Early ChatGPT 3.5 was not that useful. It was a tech demo, it hallucinated, it lied, it tried to please and what have you. What people bought into wasn't the product, but the promise of the product in the future. Integrating chatboxes into everything have failed, and even Microsoft is trying to rebrand. The product then, failed for businesses, and agents filled in the gaps.
People do not invest for the current product, they invest in the future product. And fear mongering is essentially an extremely effective signaling for making the future look bright. Keep in mind that back in early GPT 4 days, people were saying that hallucinations would be fixed in 6 months to a year (or pick a time-frame). What they meant was models not having hallucinations, what we got was agents looking up info on the web and summarizing it (and hallucinating anyway).
> "The equities here ..."
For big tech companies court cases like this are nothing but theater. Always has been. Dario will go on the senate hearing tomorrow and will plead that his AI is dangerous and governments should take the step to stop them, with the same rigor that he claimed GPT 2 was too dangerous to release openly. He might also mention distillation attacks and how open models are getting too dangerous as well.
> Here's the evidence ...
This is the where we will have to agree to disagree, if we haven't done so already by this point.
That entire thing is theater. People have anthropomorphized LLMs for a while now, and they are all too happy to do that when they see a large language model, produce language. I see no indication that anything is going rogue the same way nothing was going rogue when you could convince ChatGPT 3.5 to wipe out all humans as the context window got longer.
Agents can hack? Yes, that is very impressive. I say that without any sarcasm. It is straight out of sci-fi movies, to be perfectly honest. But agents going rogue? No. Absolutely not. Purposeful, plausibly deniable incompetence for the next sales pitch - the same one we've seen for years. Fear mongering.