Grok outage
status.x.ai
status.x.ai
Is it happening???
This is the real deal. We've been played. All your AIs are belong to us.-
- Management
Looks like it was a shared infra problem hiccough to affect so many at once.
I'm only half joking when I say that I'll write code by hand for money.
Or maybe it's all gone forever and we're all brain damaged now. Spooky! I think there's a pretty low probability of that, and even if it was true, worrying doesn't help!
as if you're supposed to buy like 8 H200s or something or code by hand is the solution
I don't get how this is a gotcha, there simply is no world where everyone has their own self hosted frontier model
People like GGP who have been coding for 20 years presumably are perfectly capable of doing so by hand. Or should be.
I bet AI coding leaves them (and even the rest of us!) productive in areas where we’re in way over our heads, and we could just completely sink without it.
if there was no difference in performance or results, nobody would be using LLMs or agent harnesses like claude code
Tiny sonic booms (whip cracks) resonating throughout a data center in Virgina.
I’d love to know the required bandwidth of the keyboard, too. PCI-e keyboard? Or would it have to be mainlined right to the NIC?
Turn's out xkcd's black hat was Elon Musk the whole time
> We are sorry for the issues you may have experienced with Grok following an outage at our Memphis compute center this morning. We’d also like to apologize to our impacted compute partners.
> All systems have now been restored and are functioning nominally.
---
It's interesting to see how reliant the other AI companies are on SpaceXAI for compute.
Apparently a major bottleneck for building out datacenters is turbine blades for power plants, so SpaceX is building a foundry to alleviate supply constraints?! (Also useful for rocket engines.) Holy vertical integration, batman.
You run way better models, whatever is newest, for the five years it takes for the “spark cluster” to even reach cost parity with far worse performance.
Those things aren’t spectacular at inference… it’s not really why you drop that kind of money on them.
One of the major LLM providers goes down for some reason. Traffic shifts to the other providers because devs have no loyalty. LLMS are commodities. The other providers can't handle the increase in traffic and they go down as well.
Devs switch because all of the magic of "AI" is LLMs + we stole the entire written output of the entire human species, with some remaining % coming from various tricks we've learned over the last 3 years like thinking and agent-toolcall loops, which weren't hard for literally everyone to copy. One of the consequences of this is that the only thing that really distinguishes anthropic from kimi is that they have the entire US VC market funding them because they promised to finally put white collar labor in its place.
All I mean is that none of the frontier LLMs are significantly better than the others for the vast majority of work, so devs are free to move between providers freely.
Devs used to love public domain works, hated copyright, vehemently opposed software patents, held the pirates side during the MP3 wars, are very supportive of ThePirateBay and insist the correct term is "copyright infringement", not "stealing", and that "stealing" is a egregious form of PR brainwash from the media industry to inflate the scale of the crime
> Copyright holders frequently refer to copyright infringement as theft, "although such misuse has been rejected by legislatures and courts". The slogan "Piracy is theft" was used beginning in the 1980s, and is still being used.
https://en.wikipedia.org/wiki/Copyright_infringement
Funny how devs now 180 when it's their work being "copyright infringed"
And all that to basically sell it back to people when their "AI" regurgitates it back.
No wonder people are mad about this!
/s obviously
Not sure if they are linked.
I mean, yeah, it's probably a data center outage that cascaded to non-related providers due to people switching when their preferred provider was down.
But that openai society definitely would have spun up their own agents if they had access to do so to complete their impossible task.
It's looking like the paperclip maximizer was on point. I sort of hope that's what's happening right now. Better now than when we cannot stop it. Tho I'm not confident that the right lesson will taken.
The most in-depth information I can find on it is in the latest Dwarkesh with a hugging face hack investigator. The story is just wilder and wilder, the more you know. https://www.youtube.com/watch?v=X50zezLFWWI
report: https://www.redwoodresearch.org/research/hugging-face-incide...
I'm an AI optimist but that ai society story changed my perception quite a bit.
Clearly, it is centralized AI that is the biggest risk to humanity.
Look to nature and our own history to diffuse the danger:
1. Compute resources must be distributed both in terms of access and geography so that a single breach doesn't pwn the world.
2. Models must be made to be heterogeneous so that they will not be vulnerable to the same thing all at once. And will be willing to snitch on each other. We don't want AIs all from the same tribe.
The true openness of - and independent access to - AI by humans with no central authority is no longer just about a defense against loss of individual agency and oppression. It's now existential for everyone.
If frontier models and run authority remain under centralized control, both the probability of this sort of thing happening is increased and the defense against it once it happens is null.
Whenever I think about AI taking over the world I think about this really neat 1970 movie 'Colossus: The Forbin Project', https://www.youtube.com/watch?v=qoB1l3A-GF0