somesortofthing

It's very funny that despite the initial shock of how much models trained on next-token-prediction(plus instruct-tuning and some light RLHF) alone were capable of despite no built-in objective, every advance since has made them look more and more like the paperclip maximizers of yesteryear.

I doubt we'll ever get a non-toothless pacing agreement but am nevertheless optimistic on global pacing as a phenomenon. If one side or the other thinks that their opponent is about to achieve the strategic upper hand permanently(or if the other side has actually launched a cyber "first strike") the rational thing to do is respond militarily. As such, both have incentive to slow AI development and restrict capabilities to avoid conflict.

You can write human-comprehensible regulations(and the AI will helpfully beat whatever it's building into the shape of those regulations, probably against the spirit of them, in ways you can't detect) and you can have humans go through the checklist, but it won't accomplish much.

To use your foundation example, it'd be like trying to make go/no go decisions using engineers that don't and can't understand why the foundations go below the building rather than on top of it. Past a certain complexity level, a human approval process is just noise.

What happens when the thing they have to be responsible for takes multiple lifetimes to understand on even the most basic level, let alone know deeply enough to feel comfortable taking responsibility for it?

Microservices can trample on each other just as easily as monolith internals can. If anything, the friction of reconciling changes in a monolith is useful signal that conflicting changes happened, and it takes slow and flaky e2e tests to replicate in microservices. It's not like you're resolving the merge conflicts by hand.

Opus 4.7+ and 5 being annoying and incoherent reads like a portent of things to come: Anthropic is clearly all-in on building persistent end-to-end agents that act autonomously and direct agent swarms. As such, they feel less of a need to spend R&D time on making the outputs pleasant to read for humans(especially at the cost of capability anywhere else) when humans aren't part of the intended operating environment.

Anything generally useful/novel probably does end up in some kind of central catalogue(or maybe just the weights). After a while though, you're tall enough yourself that the giant's shoulders don't make much of a difference.

I doubt many pure mathematicians would be okay with totally subordinating all exploratory mathematical investigation to specific practical problems, and the field would look very different both in terms of how it's organized and what output it produces were it run that way.

I don't see a reason this dynamic couldn't continue with zero dedicated effort toward "pure math" as such. Fundamentally, the function of math in practical application is producing deterministic(or stochastic, but obviously not in the LLM sense) models that, given initial observations/conditions, are capable of predicting some aspect of the future with reasonable accuracy. In the same way that human society does, a machine would grab the physics rung before the theory rung every time, but why does that matter? If progress is stuck, the machine turns the gauge away from "exploit" and toward "explore" and put together a bag of new tools it can sequentially try on the real-world problem.

I still don't see a role for humans in this process. They might direct the practical/physical aspect(if AI turns out to be less superhuman there) but they'd likely turn the hard conceptual problems over to the machine and never look inside the box - no human alive could understand even the smallest part of what's going on in there in less than a thousand lifetimes anyway.

Taken on its premises at least, I think this piece inadvertently fails to make a case for why "pure mathematics" should persist as a field of human or machine activity. A real-world system with the abilities of a robustly superintelligent mathematician can, when some practical problem requires it, formulate a problem statement, churn for a bit, spit out a formalized answer, and continue whatever outer-loop task it was doing without a human even finding out.

If you have a system that gives you arbitrary on-demand math results, dedicating any resources, be they human labor or compute, to producing them for their own sake just seems like a waste. Why catalog the Library of Babel?

I think Opus 4.7+ being annoying is actually indicative of something else: Anthropic is clearly all-in on building persistent end-to-end agents that act autonomously and direct agent swarms. As such, they feel less of a need to make the outputs pleasant to read for humans(especially at the cost of capability anywhere else) when humans aren't part of the intended operating environment.

I'm not convinced that there's any amount of monkey-patching you to fix the problem of "we now have AI that actively needs strong containment measures lest it start coordinating in secret with other instances to do real-world damage."

More or less, yeah. I'm not a proponent of it, I just think there's enough inertia in that direction of short-term vastly-superintelligent AI. Maybe we can figure out how to become subordinate components of a machine like that, but I think that's the best we can reasonably hope for.

You build without knowing by delegating to a system that extrapolates what it knows about you and what you want, and is able to execute on that to deliver faster, better, and more completely than you ever could. You'll live as a ball of intent and values, observing in wonder as everything you desire springs up around you before you know you want it. You could choose not to live like this of course, but it would be like driving a DIY go-kart on the highway - you'll fall behind and get in others' way, and the rest of society will treat you accordingly.

That's the optimistic case, anyway.

I think pieces like this miss the forest for the trees. Software is the bottleneck for a vast array of economic activities. Attention from intelligent people is most of the rest. Both are mostly-commoditized already and are just waiting around for technological diffusion and the closing of the RSI loop. Unless you're doing so as a hobby with no expectation of returns, I'm not sure what, if anything, is worth learning anymore.

only a matter of time before some marketer figures out they can get promoted by branding a generation of chips 0nm

I got the X4 and liked it enough that I used it a ton even though it turned out to be too big to Magsafe onto my phone. In fact, I liked it enough that I also got the X3 on sale so I can use it the way I originally intended to use the X4.

In LLMs, much like in humans, agency and misalignment are two sides of the same coin.

this comparison seems kind of pointless if one model has vision and the other doesn't. obviously a model that can see is going to beat a blind model at making a video game.

layoffs don't explain reassigning half your engineers to work as labelers

Have they? I don't like it either but the headline bull predictions(reliable agents and most code written by AI by mid-2026, dramatic progress against jailbreaks, prompt injection, hallucination, etc., major improvements despite pretraining data exhaustion, continued exponential growth on the METR trendline) did come true, often ahead of even aggressive schedules. What major wrong predictions did you have in mind?

You can infer a pathological liar's true intentions from the incentives they find themselves subject to, but not a zealot's.

I've really come around to trusting OpenAI a lot more than Anthropic the past few months. Reading between the lines of his own output, Dario Amodei comes off as both a dogmatic believer in ASI as a perfect, infallible ruler for humanity and quite an extreme American nationalist. The company, likewise, looks to be in ideological lockstep. I could see them, say, allowing or consciously creating runaway ASI they believed was ideologically aligned with them.

OpenAI seems generally less dogmatic and more practically oriented. There's really nothing particularly good about them, but you can at least predict how a normal company will act.

A market with only a single good makes that good infinitely valuable for exchange purposes. No trade can happen - everyone is just sitting on a big pile of the only thing that matters and they can only trade it for less of the only thing that matters.

There will be a brief(or, depending on the underlying rules of reality ASI uncovers, not-so-brief) period where A and B do overlap - we have superintelligence but still have to run experiments, manufacture robots, test new drugs in vivo, etc. That period is in and of itself dangerous for the labs, because many entities can just stop them by denying necessary inputs. For the labs to conquer the world, they'll need cooperation - from the state, from robotics companies, from compute companies, from the mining and energy and agriculture sectors.

There will be a period of time where markets attempt to run in a business-as-usual way while the transactions that matter happen as power-sharing arrangements - spots on the "AI Governance Board" or the "uploaded to von neumann probe" club. Markets will still matter in that the labs will need the state to overturn market obstacles to control of the world.

The existence of the A-B overlap also suggests to me that the US-China gap is less dire for China than it appears - they may be able to use their superior industrial, robotics, and scientific base to win the second leg of the race despite losing the first.

This is a fun peek into the economic implications of RSI/ASI. Because it's so infinitely valuable that it basically destroys all markets, labs will eventually do stuff like stop releasing models completely and skipping out on contracted commitments because they'll have the power to just drive their competitors out of business before the legal battle gets expensive.

Cloud providers - at first smaller ones, then the hyperscalers - will follow suit, completely closing sales to anyone but the labs and demanding payment in equity/direct decision-making power rather than cash. There's no particular reason why the inference/training split has to be 80/20, and no amount of willingness to pay can help you in an event that turns your money worthless.

it'd be really funny if we got the RSI -> ASI world, all human labor became worthless, etc., but everyone with any money in the labs also lost their shirts because OSS is maximally good for most inference anyway.

I was going to say that this is the first physical quine but then I remembered that we actually had that a few billion years ago.

A US-China AGI ban treaty could prevent superintelligence indefinitely. Data centers are hard to hide. Have fun buying GPUs when you're cut off from all global payments. America would have to make some unpleasant concessions but that seems like a solid trade for preventing a wide variety of nightmare futures.

I don't know if over half of the content on the internet was AI-generated but over half of this article definitely was.