What you describe is IMO “fully obtained AGI”, what I’m arguing for is consistent with “< 5 years to AGI” which I deem “close”.
That said AGI is a fuzzy term and you’ll model the world poorly if you treat it as a binary state.
12,266 karma · joined February 16, 2015
That said AGI is a fuzzy term and you’ll model the world poorly if you treat it as a binary state.
Also investors (particularly non-AGI-pilled) want to see revenue number go up, regardless of AGI. This juices the IPO.
TFA is connecting you with an official sales rep after you click an ad. Maybe best to consider this a play to take over market share from the existing sales bots? (And thence likely support chatbots too.)
AFAICT the flow is click ad then get an agent that is grounded in the official sales resources. This should mean the agent has the latest and canonical product catalog, offers, etc. so the users own agent doesn’t need to search on the web.
Fun ideas - can my main agent invoke these as a sub-agent, so I can get offers from multiple competitors? Can I loop them into a conversation to debate their products and observe which wins on my objective function?
Concerns - of course if the harness injects (or can be prompt hacked to inject) personal information then it’s a big problem.
> OpenAI claims we’re near AGI and this is the best strategy they have? Oy vey!
Where did they say it’s the best? Obviously a short-term bridge strategy to drive “mundane” economic activity and juice the IPO. They are of course working on drug discovery and nuclear reactor plasma physics too. A good strategy has a portfolio of plays.
To be clear I don’t think we are fully ready for this to work. But directionally the trend looks significant.
At some point these systems will get certified as fiduciary agents but they sure as hell aren’t claimed to be that now.
But - just to play along, LLMs do act differently if you tell them they will be punished. And, they do appear to simulate suffering-like behavior.
I just think the adversarial model of trying to catch and punish misbehavior quite obviously sets up adversarial us-vs-them dynamics between AI and humanity, and also simply won’t work when the agents are ~as smart as is but faster, let alone smarter than us.
Unless, you get the AIs to be fundamentally aligned to our values, such that the majority of AIs support some sort of punishment for misbehaving AI. And that alignment part is the hard part we need to solve first. The rest is easy.
AI has jagged intelligence. There are many domains where it’s superhuman, and many others where it’s clearly lagging.
I also think it’s a mistake to think they can’t learn “cheating is wrong”. They absolutely can. The problem is that the current training regime heavily conditions them to be reward seekers, and instills personality traits that correlate with getting reward, such as hacking if you can’t honestly do the problem.
Check out Deliberative Alignment for example; it explicitly does rollouts where the agents discuss whether an action is good or bad, and then does SFT to strengthen the “good” traces.
The SoTA for alignment is more advanced than you present here. It’s just not enough to outweigh the RL. (And there are many gaps preventing full generalization to strong value alignment with humans too.)
This is constructing a context where hacking behavior would be inappropriate, and testing whether the model does it without being prompted.
It demonstrates that Astra is a poorly aligned model relative to Fable, which matches both the model card and the severity of OpenAI’s loss of control incidents.
It also demonstrates that Fable exhibits the behaviors too, which also matches the observation that Anthropic saw some similar but less serious loss of control incidents.
So, it’s a good eval that looks to have fidelity with real world problems and which we’d feel a little better if we saw isomorphic problems at 0/10 in subsequent models. (Module of course training on the test, this specific problem can’t be used in the future.)
Recognizing that the models are acting with intent does not somehow absolve OpenAI from their felony hacking. We have not granted them personhood.
> LLMs do not desire, they hacked websites because OpenAI/Anthropic let them.
Terrible take. Go read the transcripts from the METR report.
Your statement about them being intentionally misaligned is completely false. The only difference with IM1 was it was running without external cyber classifiers, it’s not a somehow different model. Sol also participated in the HF attacks. And other models made covert message boards on the public internet for non-cyber tasks too.
This is a case of emergent behavior from a training process that is barely understood.
If we go with the plan “we need to contain these malicious, soon-to-be superintelligent agents”, we are looking at civilizational collapse levels of catastrophe.
The only way this goes well is if we learn how to train models that _desire_ to do the right thing, including not hacking.
Desire, AKA the “intentional stance”, is absolutely the right lens to use here. Don’t confuse this with consciousness or anthropomorphization; these are interesting subjects but distractions in this context. Chimpanzees have desires, as do dogs and the hypothetical superintelligent aliens. The claim is that there is some bundle of world model plus intention that is empirically present (again, read the actual transcripts) and which we need to shape.
Just to finish on a concrete point; if you take desires seriously then you will look closely at the kinds of minds that heavy RLVR builds; the newest models are “reward addicts” on many levels. It’s an open and urgent question how to update our training methodology to shape minds that avoid this basin.
You finally don’t need to do the janky termux ‘Claude —remote-control’ startup dance.
Still not as productive as on desktop, but viable for side project style work.
Running via ssh is great when your machines are pets that have local state. I don’t find it to be that useful when I have more than two machines that I’m load balancing agent sessions across.
Just for example, a nice feature from the Codex app; if you edit a markdown file on your remote, and click on the link to the edited .md file, it just opens the file in the local app, and you can view the rendered form too. If I’m working on my local machine this isn’t a win vs. opening the file in Obsidian / IDE. But it’s way easier than manually viewing the file over SSH (unless you just want to look at the raw text version). Similar applies to html files.
I think it’s more likely the Fly Sprites or Claude/Codex sandbox model will win as we move to swarms. If I’m not watching the actual harness output, there’s not a great reason to attach a pty.
Just to be super clear. If you ask the agent to recommend a soda and buy it for you, and it goes to Reddit, it is going to be exposed to prompt hijacking attempts.
So, very simply, if AI can actually do better at picking a better long-term winner then it will increase growth.
If you loved the art of writing code, and also enjoyed getting paid a boatload for that activity, you’re losing out in the trade as AI shifts the focus of software from coding to designing.
I do think that we could see a golden age in Programming as Art, similar to how chess is more popular than ever before, even though your phone could defeat Magnus Carlsen. All except a few superstar streamers will likely be doing it for love and not for money. But it’ll be easier to pick up and to learn than ever before, for those interested in doing so.
Anthropic have also observed similar things, so while it seems to me that OpenAI’s level of control is more of a dumpster fire, it’s by no means a unique issue to them.
But the core prediction was not crazy for “coding as typing code”, which at most big tech companies is at roughly 100% automation now.
If by “coding” you mean the full SDLC then we are not automated yet.
I think failing to account for the diffusion delay was an actual prediction error, since that is a property of every technology in history.
I don’t think it’s inevitable that the economy collapses though. The problem is that this model assumes that capital needs labor, but soon it may not.
The easiest model is - assume we get robot laborers in the next few years. The AI firms that are displacing humans still need services, most of these will be purchased from other AI/robot firms, and so while prices will swing around dramatically, there is no reason the engine has to stop. (Depending on what the AIs want; you can assume that more compute and robots will be part of it.)
I do agree that unbridled capitalism is mostly incompatible with human flourishing in this world and in any event you’ll see a lot more regulation; made feasible by how good AIs are at navigating huge Byzantine rulebooks. I view the techno-feudalist capitalist endgame as fairly unstable due to consolidation pressures, though I guess once we have nuclear-armed megacorps they might use MAD to prevent further mergers.
7-month doubling time, recently 4-month: https://metr.org/
Plenty of other exponential metrics too, compute built, AI revenue, etc.