Noam Shazeer Joins OpenAI | Hacker News Reader

Noam Shazeer Joins OpenAI

twitter.com

355 pointslukasgross425 comments

https://xcancel.com/NoamShazeer/status/2067400851438932297

https://www.reuters.com/technology/googles-gemini-co-lead-no...

For those interested, Wired ran a backstory about the Attention is All You Need paper 2 years ago: https://www.wired.com/story/eight-google-employees-invented-...

It gives some context on the contributions of each of the authors. About Shazeer, from the article:

Shazeer’s joining the group was critical. “These theoretical or intuitive mechanisms, like self-attention, always require very careful implementation, often by a small number of experienced ‘magicians,’ to even show any signs of life,” says Uszkoreit. Shazeer began to work his sorcery right away. He decided to write his own version of the transformer team’s code. “I took the basic idea and made the thing up myself,” he says. Occasionally he asked Kaiser questions, but mostly, he says, he “just acted on it for a while and came back and said, ‘Look, it works.’” Using what team members would later describe with words like “magic” and “alchemy” and “bells and whistles,” he had taken the system to a new level.

> Using what team members would later describe with words like “magic” and “alchemy” and “bells and whistles,”

Ok, these peopl have all gotten extensive training on how to hype for the non-technical crowd without saying anything of substance.

"Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith."

https://news.ycombinator.com/newsguidelines.html

Does that apply to quotes from an article? They seemed to be criticizing a second or third degree source for being PR, which feels fair.

Yes, in the sense that if there's nothing interesting to say about a quote then there's no reason to copy it into the thread.

As a hacker, I kinda like naom's code. I was had to implement a TC MoE kernel, and stumbled upon his code from [tensor2tensor](https://github.com/tensorflow/tensor2tensor/blob/master/tens...) and i think "alchemy" is justified. Dude writes some beautiful kernels.

He also saw LLM would replace search before anyone else, and that is something to look at the Lamda or GPT-1's output and think: yeah this will answer all of our questions one day.

Also, evaluating complicated functions with numerical stability and automatic differentiation is hard.

There's no doubt about Noam's abilities. But I read through that code, and struggle to see its 'magic' or 'alchemy'. Can you elaborate what you find especially good about that code? (You may assume GPU kernel programming knowledge on my end.)

To me the magic Noam moment was when he came to my team and said "that cluster has a bad node in it, but this other one doesn't" and we had to spend like a week tracking down a single bad processor out of thousands.

Unrelated to the particular code above. There's a difference between writing code about or adjacent to a proven idea vs writing code in uncharted territory. I suspect that is what happened here. It's the same thing with say music and art. A lot of people today can play Chuck Berry.

It's a good point. Though I do wonder if the magic he casted was more at the conceptual level (intense belief on a set of primitives that ought to work) more than the code itself. Even by 2018's standards, the Tensorflow code above doesn't really look that impressive. It's hard to judge based on those past standards, though. But, wonder if somebody who knows more than me can elaborate.

Are you saying that with today's hindsight, or would you be saying that at the time of its creation?

The "bells and whistles" label sounds more dismissive / perjorative to me. An odd, and not a particularly nice, thing to say. Makes me wonder how the "magic" and "alchemy" terms were intended in this case, also.

If I use the words alchemy and magic about a piece of code, those are not flattering words.

Wow. What could possibly have caused him to quit so soon after coming back?

I hope this is not accurate but I'm afraid it is: https://x.com/signulll/status/2067446889956430273

If I had to make a guess, money played a role lol.

He is close or already a billionaire, not sure much more money will be do much heavy-lifting

you'd be surprised! people seem to have a limitless appetite for that money stuff. they just can't get enough of it, i've found

Most are happy and stop at multimillionaires, but of course we don't hear about them. The focus on hungry billionaires is survivor bias.

We don't hear about Tom from MySpace.

How much money do you have? or are you just commenting from the peanut gallery?

i’ve got 300k in a 401k, 50k in cash and i earn 230k pretax. i’m not sure what the point of the question is tho

You're doing well, but what of you want to buy a house?

My point is you and I are still working for money because we want things/security etc.. I really don't get the sense that people who have tens or hundres of millions of dollars are doing it cause they need more money. It's other things that motivate people. I run into people in my workplace that just enjoy it, even though they're sitting on 20M.

All those engineers making 20M a year at Anthropic and OpenAI are going to back down to normal super high comp of 700k a year after their starter grants run out, and yes many will quit but the people who stay aren't moving the needle on their finances that much.

I know some pretty wealthy people. They are very aware of those who are 10x wealthier than them. If Noam has 1B, he is probably pretty aware of those that have 10B. He's met them and seen their properties, scope, and powers. Likewise, they are thinking about those that have 100B, and those are thinking about Elon, who now has "four commas."

That's not really Noam's style

For many business people, money is just a measure of status after becoming rich.

Maybe Noam measures status differently.

He left Character.ai for money.

signull is more of an anonymous sh*tposter than a known industry insider, but I think this does capture the sama contribution to OpenAI very well. At least from an outsider who follows this stuff based on vibes.

That twitter story isn't anything unique to OpenAI or Google, it's just classic "big public corp vs private startup" culture. Once you have to worry about the SEC, shareholders, antitrust, regulations, lawsuits, etc. it's very, very difficult to avoid turning into "big corp" culture.

Sama, and any other founder, will always have a difficult fight against bureaucracy, and once you let a little bit in, the bureaucracy's sole purpose becomes to grow itself.

Google and Apple both need a culling similar to what Elon did with Twitter after taking over.

I disagree. It's not about the culling, it has never been, and actually, it makes things worse. You spend countless hours and tons of money recruiting talented people not to lay them off because you don't want a bureaucratic org.

If the issue is inefficiency, tons of meetings, too much team alignment etc, then that's the issue that you need to tackle, and these issues can already appear in a 50-100 employee company. Sure, that's an easy problem to solve with a smaller size but unless you hired people for no reason, these people have a very specific set of problems to tackle and are often, in these companies, the best in class to tackle them, culling half of the company isn't going to make things better.

(And X rehired part of the laid-off engineers)

That depends who you are firing. There are many job roles who's primary output is meetings and documents.

What percentage of Google employees are engineers...

Google bloat gave us transformers. Apple bloat gave us a usable touchscreen only, pocket computer (famously an entire org within Apple had developed an iPod-based approach that was competing with what was released)

The leaps forward need bloat. A startup can execute on specific vector direction way better.

Now back to your point, what did X deliver with its lean ops? It seems that it needed 2 bailouts (one from xAI, and one from space X)

innovation requires monopoly is a Theil one liner -- probably true too

Probably not true

> Google and Apple both need a culling similar to what Elon did with Twitter after taking over.

You could cut Google's size by 40% and they'd still have more corporate employees than Apple.

(Google has ~190k employees, Apple has ~160k but 50k of those are retail staff, so ~110k corporate)

Google is competing with nvidia (TPU), AWS (GCP), Netflix (youtube), Tesla (waymo self driving), OpenAI (Gemini), Microsoft (Workspace), Apple (Android)....

Apple was doing effectively everything on that list, though the self driving car was cancelled and the AI is now gemini.

Err, maybe some of those, but there's a reason Apple is using GCP and TPUs for training their custom gemini model.

Is that you Musk? Twitter lost half its revenue, more than 80% of its valuation.

That was mostly because Elon told advertisers to fuck themselves at an advertising convention no?

It's impossible to disambiguate but advertiser tools, brand safety, targeting, reporting etc all need a lot of ongoing effort. If it gets harder to advertise effectively on Twitter, those dollars can very easily go elsewhere.

Eh, what has X/Twitter delivered since the cull? It’s basically in maintenance mode. Which is fine if that’s what you want to do, but Google and Apple definitely don’t (and I’m glad for that)

He merged it with SpaceX, quickly created a few colossal datacenters for training runs and somehow turned that into a 3T company.

That has nothing to do with culling staff at Twitter though.

> It’s basically in maintenance mode

Has been in more of a maintenance mode with a multiple of those people. If anything, the pace of the product has improved. Regardless of what you think about Musk, the company he bought was a bloated mess.

Features aside, it was making considerably more money before Musk’s cull.

Working recommended feed

"what Elon did with Twitter after taking over."

You mean fire the very smart people who designed the core systems AND insult them so that anyone with options would never want to work there?

Google is facing a legitimate innovators dilemma here. It makes sense to have all this process when youre protecting a $4.5 trillion golden goose. The tragedy here is that one predictable outcome of this situation is google deciding to considerably cut research funding when they figure out it just serves to bootstrap future competitors.

This is when it makes sense to split your business up into multiple smaller businesses. The government should be doing this via anti-trust but they have dropped the ball there so, at this point, the corps really need to just do it to themselves to better compete.

Wasn't that what the whole Alphabet re-org was supposed to do?

Alphabet has Google with 99% of the profit through Ads, Search, Cloud, Gmail, Youtube etc

and tens of losing companies that make balloons or whatnot

Or maybe just have your R&D teams focused on doing R&D with zero corporate interference. Staff it with personal assistants whose only job is to ensure the researchers have whatever they need and are never bothered with meetings or other corporate shenanigans. The assistants could then be the proxies to management to provide feedback to management, but only on best effort and still staying the fuck out of the way of the researchers.

Easy peasy!

Easy peasy yet still impossible for them to do as R&D doesn't add to the bottom line, it is a cost center.

Easey peasy was said with a wink, although technically speaking it would be easy -- if they didn't have their heads up their ass.

A dear friend of mine was doing R&D at a startup where management poured on a bunch of low-level management work that took him away from his joy, which was developing new tools and approaches to solving customer problems. He left, and went to another gig that promised him complete freedom to invent and discover. That job is at a Fortune 500 company that is slowly starting to pull its head out of its ass.

going to go with "money" and a lot of BS from altman

https://nitter.net/signulll/status/2067446889956430273 for those who don't want to click the above

Thank you, I quit twitter 8 years ago and generally avoid twitter links.

Someone should write a bot to do this automatically. What is the HN policy on bots?

there is the extension libredirect that automatically can redirect to nitter

That was hilarious and sad.

This reads like an episode of Silicon Valley. I wish that show was rebooted, they'd have so much funny material nowadays.

Gilfoyle was really ahead of the times with Son of Anton.

Your dream may be only a prompt away.

I loved that show. The love that went into it really shows.

Sadly the gap between reality and satire has shrunk.

But yes. I also wish that show would come back.

Noam shazeer would be google head dreamer

The gap between reality and satire was apparently already very small back when the the show was written. The creator, Mike Judge (who also created Beavis & Butthead, and Idiocracy) had worked in Silicon Valley as a developer and based the show on what he saw. Apparently it was very popular with SV insiders precisely because it was so accurate.

And office space!

Judge also consulted with various teams at places like Google; I worked with one of the guys who provided details that later showed up on the show (as well as many plushies). He didn't watch the show because "it hit too close to home"

I think real life has far eclipsed the absurdity of the original show. They might have a hard time competing with just the news now days.

Or they might give tech companies more ideas!

Even back then Mike Judge said he had to tone down the absurdity he saw on fact-finding trips to Bay Area. He said no one would believe how absolutely stupid so much of all of it he saw was.

Three comma guy would now be four comma guy

Wonder what the modern version of the hotdog app would be?

Agent harness?

Thanks for this. I will be thinking about “we can create a permission working group” for a long time.

I’m convinced this is 90% of the way this actually played out.

Noam Shazeer was one of the lead authors of the seminal paper "Attention Is All You Need", which introduced the transformer architecture. (From Wikipedia)

This understates his criticality. The author list was randomized, but the critical idea was truly his. Wonder what this says about GDM …

Even more important, I wonder what it says about HBW...

Even if we knew, we’d still fail to understand GHO

But more importantly the impact this has on TLAs

Is this a generally well known thing?

Nope, but it’s not particularly unknown either. It shouldn’t be a surprise; he had remarkable research contributions before and after (separately, he was also an IMO gold medalist).

Source for this? The notion of attention dates to a content-addressable lookup during sequence alignment (as well as, concurrently, memory lookups in neural Turing machines). Attention had been used in other models, like GRUs and LSTMs with attention. The Vaswani et. al. paper did not introduce attention, just removed everything _but_ attention (and FFW) from the network. Are you claiming the "critical idea" of removing the GRU and LSTM parts and just keeping attention was "truly" Noam's?

At some point in late 2017 the paper was updated with this additional detail:

    Equal contribution. Listing order is random. Jakob proposed replacing RNNs with self-attention and started the effort to evaluate this idea. Ashish, with Illia, designed and implemented the first Transformer models and has been crucially involved in every aspect of this work. Noam proposed scaled dot-product attention, multi-head attention and the parameter-free position representation and became the other person involved in nearly every detail. Niki designed, implemented, tuned and evaluated countless model variants in our original codebase and tensor2tensor. Llion also experimented with novel model variants, was responsible for our initial codebase, and efficient inference and visualizations. Lukasz and Aidan spent countless long days designing various parts of and implementing tensor2tensor, replacing our earlier codebase, greatly improving results and massively accelerating our research.

In any case, if the authors considered their contributions equal, that's good enough for me.

Thanks - wanted to point to this, and indeed should have worded my claim more precisely. And yes, am aware of prior work on attention. (I need to look it up, but I recall Noam saying publicly that he wouldn’t have agreed to random ordering of contributions if he knew this was going to be this big).

I don't know we can just say things now. Ah we're on the internet

The architecture was Shazeer's, but the rough idea came from Jakob Uszkoreit who initiated the project.

Uszkoreit wanted to build a more efficient/scalable language/seq2seq model that could take advantage of GPU parallelism (replacing RNNs which were the main approach to sequence modelling at that time).

Uszkoreit's insight was that although language appears sequential, it is in fact really part parallel part hierarchical, as can be seen by linguist's sentence parse trees where at each level there is parallelism/independence between the branches of the tree, with them getting combined at the next level up. This is what gave rise to the idea of a model that consisted of a stack of of parallel processing layers (transformer layers). I believe that attention was also part of the plan from day one, as this had already been proven to be valuable (Bahdanau) with RNN seq2seq modelling.

So, this is what Uszkoreit wanted to build, but by his own account he failed to come up with an implementation that matched or outperformed the prevailing RNN approach that he wanted to replace. At this point, Uszkoreit mentioned the idea to Shazeer, who got on board and eventually arrived at a performant architecture which was then pared back by an ablation process resulting in the initial encoder-decoder Transformer architecture. Shazeer later came up with the mixture-of-experts architecture, and also other optimizations after he left to found character.ai

Can you expound on the ablation process? Is that referring to a stripping down of the data or weights or something? Or a stripping down of the transformer architecture structurally? Just curious

You train the model then do a baseline evaluation. Then you evaluate many variants where you have removed or nulled out different layers or chunks of the model. By comparing the performance of those mutated models to the baseline you can learn a lot about the model. What parts don't have much value and can be removed, the location of "functions" or "facts." Etc. Google it.

How come they didn’t ablate encoder? OpenAI GOT models are decoder only.

If you read the Wired article linked elsewhere on this thread, then it explains that. The work was being done by people from the Google Translate team.

It was originally built as a general purpose sequence-to-sequence (seq2seq) model.

The research history leading up to this was interesting - there had been a bunch of work, in various domains, on "autoencoder" architectures used to learn compact representations for things like dimensionality reduction and sequence representation. The idea was to have an encoder-decoder pair, connected by a limited bottleneck representation, with the training goal of the decoder reconstructing the encoder input from the bottleneck representation.

One example of this was to learn a fixed size(!) sequence (e.g. sentence) representation using an LSTM-based autoencoder (LSTM->embedding->LSTM), which at the time seemed rather shocking - the ability to represent a variable length sequence with a fixed size embedding. Equally shocking was that you could use this for machine translation simply by connecting an LSTM encoder for one language to an LSTM decoder for another language.

This type of LSTM->LSTM seq2seq encode-decode architecture for machine translation was then improved by Bahdanau by replacing the fixed size representation with an attention mechanism so the decoder could learn to be more specific about input-output relationships.

This type of LSTM-based seq2seq encode-decode architecture, using attention, is what Uszkoreit et al set out to improve - to make more efficient by using a parallel vs sequential (RNN) architecture. The Transformer was never conceived of as purely for language modelling, or as an "AI" architecture. Later when the usage focused on language modelling (generation, not translation), the encoder was dropped since input and output are the same thing.

Curious about others' contributions, such as Vaswani, Parmar, Jones and Gomez, to the paper. What sucks about co-authorship in research papers is that you don't get a clean breakdown of who contributed what to the research paper, and the distribution (in more cases than not) is very much like a pareto distribution.

I'm talking from plenty of group project experience here.

> What sucks about co-authorship in research papers is that you don't get a clean breakdown of who contributed what to the research paper

Why? If you read a research paper for its content this is not especially important.

This thread is more about the people of course, and here we care, but that's not the point of a research paper.

This is fascinating. Do you know if there's something I can read that has this mix of timeline and technical detail?

There's an interview with Uszkoreit here that gets into a lot of the development history.

https://www.youtube.com/watch?v=ayBOXv_SFCY&t=164s

That’s not true. Jakob, Ashish and Ilia for the core idea and initial implementation and Noam for several critical details on implementation.

Some context for people who haven’t followed the full loop: Shazeer was a long-time Google researcher, joined Google in 2000, and was one of the co-authors of “Attention Is All You Need.”

He left Google in 2021 to co-found Character.AI. In 2024, Google brought him and some Character.AI researchers back via a licensing/talent deal with Character.AI (reportedly around $2.7B). He was then made a Gemini co-lead.

Now he’s leaving Google again for OpenAI.

Exciting times!

this character.ai? https://www.bbc.co.uk/news/articles/ce3xgwyywe4o

The Netflix documentary will reveal he was secretly working for Sam Altman the whole time... (Cue diabolical VC-backed evil laugh.)

Google lost three critical years chasing AGI, and got acquired by SpaceX, now a Dyson Sphere startup whose pitch deck is just: "What if we put a paywall around the Sun?"

Oui!

I first saw Noam on Dwarkesh’s podcast together with Jeff Dean. Recommend if you want a taste of what’s Google’s folks take on things.

https://youtu.be/v0gjI__RyCY?is=nz77XP4KiJy7L1AX

> Exciting times!

What is exiting about this?

Maybe he figured out a good way to short AI companies?

Right?! Unless you think this move is going to generate general excitement in our lives, it's just another rich guy moving from one high paying job to another.

At this point is it even pay that’s tempting or is it more about what they get to do? I would assume Google could easily pay them what openAI can, unless as an older company it’s harder for Google to match something really out there

Yeah my current feeling is that once I had double digit millions earning further money would be pretty meaningless to me, and the difference between 'large salary' and 'even larger salary' would be even more meaningless, but who knows maybe it really would change me. I kind of assume people like this are primarily chasing the most interesting/impactful work though.

The problem with this belief is that it implies that all of bigtech is massively overpaying for top talent who would happily stay on for pennies. While bigtech overpaying talent is more plausible than any other bigcorp doing so, it's still rather unlikely.

Only if there was a cartel which would agree to never outbid each other, of course... You can ask Steve Jobs about that one.

I think money’s marginal utility just changes from a vehicle for material comfort to a way to keep score.

It gets to the point where what you do is the main question while payment is barely a minor concern way earlier than that point, at least in my experience. You don't need to be in the top AI research tier for that.

Hopefully will get to the conclusion that "Hopfield Networks is All You Need"

How can an acquired dude leave after less than 2 years?

OpenAI pays for the earn out he would’ve otherwise received at Google + a new comp package. Made up numbers, if Google still owed him $10M for lasting the full two years, OpenAI can just pay him market rate +$10M.

Yes, but what about the audacity of it? Get paid a lot to join a company but then decide to get up and leave again 2 years later? He just wants to be passed around?

You can have character and be loyal to Google (lol) or make $xx million… I’m not surprised when people choose the latter.

I would argue its not the millions though, but rather that sweet rare compute - OpenAI has more of it for his interests than anyone - it is understandable why an exceptional mind would prioritize access to greater capabilities above all else

Oh my goodness think of the poor multi-trillion dollar company!! No honour among thieves these days...

There's a possibility that he lost out in internal political battles, and things weren't going his way. Google is full of battle-hardened political warriors who will do anything (subterfuge, sabotage, etc.) to win battles. It is possible that a guy who just wants to build cool shit would feel like a misfit in such an environment.

But he worked there previously and is Intelligent so knew all that before rejoining?

True. But earlier he had the ear of people like @jeff ; but now even Jeff has been sidelined a little (Google Brain is no longer under Jeff, AFAIK). The MBA types can be brutal and since they're not technical, they don't have the same level of respect and deference for Noam.

Google is a different place today than even 5 years ago.

$$$$

Noam Shazeer is not some super brain anymore.

His raw skills are likely atrophied by now from delegating and operating at higher levels. His entire value is mostly political now.

If he can’t play the political game and win, he’s useless. He would probably just go to some other organization where people aren’t immune to his lore.

For context, the reason he left Google the first time was because Google wouldn't ship the chatbot-type products that he saw were possible.

Google bought him back (with lots of money) and made him one of the leads of Gemini.

Looks like Google is leaking both AI talent and know-how something fierce ... and since the very day the transformer paper was written.

As an outsider, I'd be really curious to understand why, given how well positioned they seem to be in the AI battle:

- huge, quasi unmatched data war chest

- huge, quasi unmatched, planet-scale infrastructure

- native AI chip design and production (TPU)

- the core ideas for what we now know as "AI" were invented there

- deepmind, enough said

- pretty much the deepest pocket of all the AI players with the possible exception of MSFT

- a massively large user base and reach to deploy AI to (Android, YT, Cloud, Search, Email, ...)

- supposedly one the best engineering culture of the valley

Why do the best people leave ?

Why do their AI product always come in 3rd place ?

Why can't they seem to take the lead, both in terms of product design or in term of raw LLM performance?

The only answer I can think of is:

- culture is completely broken

- management sucks something fierce

- company is so fat and rich no one is actually interested in winning anymore

Google has muddied the waters on their Gemini usage statistics as it now powers a big chunk of Search. Depending on how you cut it, Gemini (and Gemini powered products) are probably producing the most output tokens seen by the most human eyeballs by a large margin.

Google at its core is not a dev tools company and it has become evident that is where the money is given the verifiable nature of software. Hixie's reflections on his tenure at Google still ring in my head to this day, though I have never worked there[1].

The people at the helm of Google no longer see the company's identity as something which must be channeled through a product or an experience. Some will point to the DoubleClick acquisition, others will point to Google Reader, or Pichai's ascension. Despite his very short tenure, MBA/McKinsey-brain is a very real phenomenon and it's no mistake that it shaped the "promotion packaged as a product launch" culture that steered Google away from seriously betting on anything that wasn't ads. To quote the signull tweet linked elsewhere in this thread, you can have everything at Google, except for permission.

Most importantly--I don't think there's a single tech product where I can point and say "Google wouldn't do that". You can contrast this with say, other Alphabet companies which don't suffer from this remotely as much. It is VERY clear what Waymo and YouTube are trying to accomplish, and while it frequently makes a ton sense for the companies to share infrastructure and product knowledge, YouTube does an exceptional job on the product side of making it very clear what they would and wouldn't do. They have experimented and shut down experimental features before (is their MOOC functionality still around?), but since it's fairly clear Google specifically is no longer working in service to the mission of providing the world's best digital portal for accessing information, I think it would behoove of them to figure out what their mission is.

1: https://ln.hixie.ch/?start=1700627373&count=1

Sounds like Noam just wanted to serve 5 terabytes.

https://www.youtube.com/watch?v=3t6L-FlfeaI

Wow - Google paid a couple billion dollars to bring Noam back. Really impressive by OAI if this reporting is accurate!

It is accurate. Confirmed by Noam himself on X https://x.com/i/status/2067400851438932297

Love the choice of words by Noam- exceptional team for OpenAI, amazing team for GDM.

Love this type of detailed textual analysis.

it would sound weird to say either word twice in such a short blurb.

[Edit: note that my comment was reparented, it was originally a response to someone claiming Noam was another "Scam Altman". I don't mind the reparenting or the killing of the original subthread, but I feel like this is necessary context to understand this.]

Noam is the real deal, he was pretty legendary within old-time ('00s) Google engineering. Paul Buchheit had a story about interviewing him with the "how to write a spellchecker" question and then him coming up with something better than the state-of-the-art, then basically delivering Google's spell corrector in his first 2-week Noogler project.

https://www.youtube.com/watch?v=gilk-76W9rE&t=60

If he is supposedly extremely smart, then surely he would have known what he was doing. So how can anyone claim all this was just an accident?

"Google and Character.AI agree to settle lawsuits over teen suicides" - https://www.axios.com/2026/01/07/google-character-ai-lawsuit...

Be aware...very disturbing: https://www.judiciary.senate.gov/imo/media/doc/e2e8fc50-a9ac...

Is this genuinely confusing for people? He helped invent the transformer, he didn’t solve content moderation.

he didn’t solve content moderation.

Considering what character.ai is, maybe he should have at least taken a shot at it.

Just from reading the threads here it seems readily apparent that he then went to start this company that did these bad things. Does not seem confusing at all?

What bad things?

Causing children to commit suicide

gotta break some eggs

Hope you are being sarcastic

Wow, he was using AI to solve problems in 2000 already, that spell corrector being trained on the Web and becoming the first widely used AI tool. Decades ahead.

Surprised to not see more comments on this, especially given the popularity of the Anthropic/Karpathy article. What a win for OpenAI - and what a loss for Google, just 2 years after paying $2.7bn to bring Noam back into the fold. Does not bode well for Gemini long-term... Or could be a signal for how deeply they are leaning into world models.

I think nobody they acquired from Character.AI is at Google anymore.

This does suck for Google. Noam will take a lot of Google trade secrets with him to OpenAi. Google's bench is deeper than this one guy though.

Trade secrets? Like how to invent a trillion dollar technology and then sit on it for years while others eat your lunch with it? Like how to consistenly release inferior quality models to others despite infinite compute and engineering talent and insane profitability in your legacy businesses?

Not really sure what you're talking about. Apple just licensed Gemini for Siri, Google and their TPU hardware is starting to hit primetime audiences that OpenAI can only dream of.

My favourite part of their strategy here is its profitability

The Apple deal is a good sign. Before that, it was like yeah Gemini might have the lowest cost per query but that doesn't matter when its users largely aren't paying for it and never will.

Google only won the deal because they have great cards in their hand. Apple's homemade PCC nodes were never going to scale, but the TPUs would. Investors expected market displacement in all the wrong places.

The tea leaves were pretty easy to read, too; LLMs will eventually be commoditized. Once that happens, the only road to profitable AI will be paved with 1) branding and 2) cheap compute. Apple does not possess cheap inference hardware whatsoever. Their GPUs are raster-focused, inefficient and expensive for their dedicated compute role. Unless Apple crawls back to Nvidia with their tail between their legs, TPU inference was their only real option. The deal was even more obvious when you used Apple's local foundation models; they're downright eclipsed by the quality of Gemma. Once Apple Intelligence was announced, it became clear that Apple was the branding component in search of a strange bedfellow to thumb their nose at Nvidia.

Would expect LLM training/inference infra to become commoditized before LLMs do.

There are multiple top-tier LLMs on the market right now. The competition is intense and the product differentiation is minuscule in many cases. Claude Code had a half-life of 1 or 2 months before getting sherlocked by the community.

We don't really see this type of opportunism in the hardware space. Case in point, the TPU is a pretty big accomplishment that successfully competes for inference and stimulates the need for cheaper compute.

https://old.reddit.com/r/singularity/comments/1u8xc9m/most_l...

Seems like there are some insights here!

edit: it seems the post has been removed but comments are viewable.

1 liner summary:

To put it lightly, the dude was politically outspoken and held strong beliefs.

Alright. OpenAI feels like a better fit for him after all

https://www.yahoo.com/news/articles/google-cracks-down-posts... seems to have some context

Shazeer is an aggressive Zionist, and while Altman is better at reading the room, he has previously aligned himself with Israel: https://www.timesofisrael.com/openais-sam-altman-says-israel...

What does Zionist mean when Israel has existed as a Jewish state for 78 years? I'm genuinely asking because the way the word is used doesn't make sense to me. There aren't similar terms for other countries to just stay the same, like for China to keep being run by the CCP. Every other country is assumed to have ontological inertia except for Israel.

I'm confused, is 78 years a long time? even the US is considered a toddler by empirical terms. zionism wasn't a thing until a minority group had the loudest voice in the room when the allies were discussing what to do with all the european refugees after ww2, and it happened to align well with the brits abandoning their failed colony in the region due to disputes with the locals

here's a quote from wikipedia. it was an utter land grab and an easy way out of responsibility for those in power

> The League of Nations gave Britain mandatory power over Palestine in 1922. British rule and Arab efforts to prevent Jewish migration led to growing violence between Arabs and Jews, causing the British to announce its intention to terminate the Mandate in 1947. The UN General Assembly recommended partitioning Palestine into two states: Arab and Jewish. However, the situation deteriorated into a civil war. The Arabs rejected the Partition Plan, the Jews ostensibly accepted it, declaring the independence of the State of Israel in May 1948 upon the end of the British mandate. Nearby Arab countries invaded Palestine, Israel not only prevailed, but conquered more territory than envisioned by the Partition Plan. During the war, 700,000, or about 80% of all Palestinians fled or were driven out of territory Israel conquered and were not allowed to return, an event known as the Nakba (Arabic for 'catastrophe') to Palestinians. Starting in the late 1940s and continuing for decades, about 850,000 Jews from the Arab world immigrated ("made Aliyah") to Israel.

https://en.wikipedia.org/wiki/History_of_Palestine

Yes, this is the important thing to know. I've heard way too many conversations that go back and forth about every act of vengeance in either direction after this, it's all noise. Partition plan started this. But I wouldn't call it an easy way out of responsibility; UK's leaders took a clear and binding position in favor of Zionism.

Also, it was Ottoman territory for hundreds of years up to WWI. I've had friends tell me for some reason about how Palestine was an independent country before... literally wasn't.

You didn't actually answer my question. How does using the word for people who want to create a Jewish state make sense when a Jewish state has existed for 78 years?

One reasonable possibility is they're referring to people like Ben-Gvir who have themselves claimed that Zionism means fighting for Israeli control over more territory like the West Bank. They're the ones calling the shots right now. I don't know whether Zionists 78 years ago would've agreed, it's possible.

To some it still means favoring any existence of a Jewish state. The inertia isn't there because aside from the original partition plan being pushed by the UK, other countries have attacked Israel several times later in ways they would've have withstood without outside support.

"Zionism means fighting for Israeli control over more territory like the West Bank."

Now that is a valid use of the term. I think the problem it that Zionism means so many different things it is nearly useless as a description. It seems more useful as a slur which has become very common in some circles.

"The inertia isn't there"

I'm not sure what you mean. Are you saying Israel could be defeated without US assistance?

I think it's valid to use the word the way that Israel's present leadership is using it.

> Are you saying Israel could be defeated without US assistance?

US and UK, yes. Not just cause of the weapons and money to Israel. After them, the top recipients of US foreign aid in the area are the bordering countries Egypt and Jordan, so that they don't attack.

> I think it's valid to use the word the way that Israel's present leadership is using it.

And how is that?

Israel has a population of 10 million people and a very modern military and nuclear weapons. If it's existence was ever truly threatened things would get VERY ugly.

Because that's what matters. The original Zionists aren't alive to ask what they think. Self-proclaimed Zionists are taking the West Bank and Gaza. In fact they've been kinda doing it for decades under previous governments, but more slowly. If there's some other kind of Zionism around, the most it's doing is complaining, and it's been outvoted.

I have doubts about their ability to self-defend because otherwise we wouldn't be giving so much money, the situation would be stable. Even if they can severely hurt the attackers, it doesn't really matter if the attackers stop at nothing. We just lost a war against Iran despite having full air superiority and killing their leader. And especially if you're considering the scenario where Israel never got Western support, and thus never got those advanced weapons.

Israel actually LEFT Gaza. You seem biased to the point of just plain lying.

They did not. They're still occupying a buffer zone, have broken the ceasefire countless times, and are blocking people from crossing.

Israel razed Gaza to the ground. It was a genocide.

Israel left Gaza in 2005.stop telling obvious lies. Hamas attacked Israel on Oct 7 2023 killing at least 800 civilians in an act of incredibly bloodthirsty barbarism, including children, the elderly, and 364 victims attending the Nova music festival. Remember when Hamas paraded the body of that young German woman Shani Louk they killed like a hunting trophy?

> Israel left Gaza in 2005

You're absolutely right, and this is antisemitic propaganda. Israel left more than 20 years ago and didn't return until 2023 due to Hamas terrorism.

They certainly didn't return:

- for five months in 2006 for Operation Summer Rains

- or again in 2006 for Operation Autumn Clouds

- or again in 2008 for Operation Hot Winter

- or again in 2008 into 2009 for Operation Cast Lead

- or again in 2012 for Operation Pillar of Defense

- or again in 2014 for Operation Protective Edge

- or again in 2018 and 2019 for incursions and special-forces actions, like that covert IDF operation in Khan Younis that got fucked up and led to a firefight deep in Gaza (but that couldn't have happened, because they left back in 2005, right?)

- or lastly, before October 7, in 2021 in Operation Guardians of the Walls.

None of those large named operations could have happened, let alone anything smaller than named operations, like special forces or commando raids, because those things are purely Hamas propaganda, and not formal IDF operations.

Right?

This is what makes talking about Israel so exhausting. Everyone of those operations was a repsponse to some violence Hamas did and you just ignore it because you are so incredibly biased.

Hamas thinks it can destroy Israel with force. It can't anymore than native Americans can destroy the US. And Hamas trying and failing over and over and over and over has made the lives of Palastinians much worse. Hamas gleefully kills any Palestinians that point this out or oppose them in any way.

Just for comparison after Germany lost WW2 they lost 25% of their land and 14 million German living on it were expelled. Has Germany spent the last 78 years trying to get it back? No. Instead they have been doing the smarter option which is creating a peaceful and rich country, which is what the Palastinians should have done.

> Everyone of those operations was a repsponse to some violence Hamas did and you just ignore it because you are so incredibly biased.

You're right. It -is- exhausting. Because I never said a word about the merit, or lack thereof, of Israel's actions or reactions, or Hamas'.

I just commented that Israel has spent multiple years in Gaza "since they left".

HOWEVER, you absolutely stated, and get bent out of shape multiple times in this thread alone, at anyone even hinting Israel set foot in Gaza since 2005:

Someone comments saying this exact thing, "Israel has been operating in buffer zones", etc., etc. and you?

"Israel left... Stop telling obvious lies."

Also, it might not be permanent occupation but when you're back there nearly every year for 3-9 months, it might not feel like you ever really left.

So Israel left Gaza and then Hamas promptly took power and has been attacking Israel ever since, and eagerly killing any Palestinian who opposes them.

Don’t wrestle with pigs or argue with Zionists about Israel (though the former will at least avoid accusing you of things you’re not).

So Zionists are pigs?

Are Nazis pigs?

So all Zionists are Nazis?

Zionism and Nazism are very similar.

It's true that Nazism lacks the religious aspect, and Zionism lacks the pagan aspect.

I hope you can agree that these are aesthetic differences.

Both movements advocate for one ethnicity having the right to live on a particular part of the land, ime. ethonationalism (Nazis: "Blood and soil" vs. Zios: "The chosen people who god promised the land to"), both movements are expansionist (N: "Lebensraum" vs. Z: "Buffer zones" and "Greater Israel"), both movements subjugate another ethnicity that they deem lesser and evil (N: Jews, Black people, Roma, vs. Z: Palestinians, Arabs).

Both movements have commited genocide against the other ethnicities.

So, are Nazis pigs? If they are, then the label fits Zionists as well.

Arabs make up 20% of the population of Israel. You people always lie about Israel. Hamas and Iran and Hezbollah want to destroy Israel in the name of Islam so only Muslims can live on the land. You people always ignore that.

"You people"? Is your paranoia getting to you?

20% is an interesting number, though. The percentage of Palestinians who were Arabs in 1918 was around 90%. I wonder if that might have something to do with why the Arabs are upset?

Oh right, Israel _has_ destroyed nearly all the entire Arab society that lived in Palestine. As opposed to your claim that "Hamas and Iran and Hezbollah _want_ to destroy Israel in the name of Islam".

You think a hypothetical backlash to an ethnic cleansing is worse than the ethnic cleansing itself?

It seems you're the one who seems to be ignoring the factual crimes against humanity that Israel commits, and only care about imagined ones that Israel would be victim to.

Reading comprehension isn’t your strong suit; personally have a preference for pigs since they’re benign and good-natured creatures.

West Bank and Gaza are different situations. West Bank settlements have been popping up continuously. Settlers were in Gaza until they exited in 2005, but now that Israel's military occupies it again, Zionists believe it should be resettled. Sorry for CNN link, but it has direct quotes https://www.cnn.com/2024/01/17/middleeast/israel-far-right-g... To be clear they haven't done it yet, aside from some illegal (by Israeli law) attempts, but they're trying.

It's pretty obvious from the emotional response that you've got some kind of horse in the Israel-Hamas war that I don't, which is fine, but I'm not gonna get called a liar too. So bye.

  My only emotion is exhaustion at hearing the same lies told over and over.

Let's not forget that while some people like to point to "From the River (Jordan) to the (Red) Sea" as some "gotcha!" that some Palestians want to "exterminate" Israel...

it was actually Likud's official election slogan in the 70s and 80s just as ... oh, let me check, Netanyahu, was getting involved in all of this, formally becoming Likud's leader in 1993.

Then please explain in your own words what "From the River (Jordan) to the (Red) Sea" means?

And the destruction of Israel is the explicit goal of Hamas as stated in their charter.

"Israel will exist and will continue to exist until Islam will obliterate it" and called for holy war to establish an Islamic state across historic Palestine.

And surely you’ll reply with what Likud meant.

Look, to be very clear: Hamas is absolutely a terrorist organization who has caused constant harm to both Israelis and the average Palestinian they purport to represent (there have been no elections since 2006, Hamas are the only people with weapons in Gaza, and the average citizen might as well be a hostage - and frankly they’d turn against Hamas a lot more if Israel didn’t steadily inflict collective punishment against them like turning off their electricity or even drinking water for days or weeks for actions of Hamas. All that is doing is making sympathizers of them).

I have no idea what likud meant and don't understand why it matters.

IMO people just use the term to mean “pro-Israel” rather than in any reference to the original meaning ("supporter of the idea of a Jewish state"). Which could mean any combination of “pro-American financial support for Israel”, “moral support for Israel in their various military actions”, “opposed to the creation of a Palestinian state”, “a belief that Israel should continue to exist as a Jewish state”, and so on. It's more about the broad political alignment than the specific meaning of the word.

Thank you for actually answering my question. That is very vague and explains why I find the word so annoying.

Zionist does have a specific meaning. It means you think the Jewish people have a god-given right to the Palestinian land, and that other creeds and ethnicities should be second class within the Jewish state in Palestine.

A non-zionist Israel would be one where all peoples had the same right, e.g.

"It means you think the Jewish people have a god-given right to the Palestinian land"

It was never actually Palastinian land. It was Jewish land, then Roman land, then Ottoman land, then British land, then Jewish land after Palastinians attacked Israel and lost. At no point were the Palastinians ever a sovereign country and in fact they incredibly foolishly rejected the UN offer for one.

"other creeds and ethnicities should be second class "

Approximately 2.5 to 2.6 million non-Jews live in Israel, comprising about 25% to 26% of the country's total population. This is compared to less than 1% of the population of Gaza being non-muslim.

Ok, well, you seem to be a Zionist. And not very educated on the matter (or willfully misrepresenting things).

The Palestinians are the people who lived there. The Zionists expelled half of them and razed 500 villages to the ground in 1948. It was an ethnic cleansing. They denied them the right to return.

There was a Palestinian identity and there was a Palestinian society. They revolted against the Ottomans, and the British promised them sovereignty. The British betrayed them and caved to the Zionists, and the rest is settler colonialism and apartheid.

> And not very educated on the matter (or willfully misrepresenting things).

Right, like stomping around the comments claiming that anything less than pretending Israel didn't set foot in Gaza between 2025 and October 7, 2023 is filthy Hamas propaganda, the existence of at least six formally named IDF operations being just a pesky bit of reality that can be easily run over by a Merkava Mk V main battle tank.

What do you think that proves? It only proves how pointlessly violent Hamas is.

Maybe it will stop you from yelling at people that they are liars when they say anything about Israel being in Gaza?

Or, who am I kidding, of course it won't.

IDF actions in response to Hamas attacks are not the same as permanent occupation of Gaza. Israel left Gaza in 2005 and Hamas has been in power ever since and has gleefully killed any Palestinians opposing them.

I'm not a zionist I think the Palastinians were fools to refuse the UN offer of their own country and attack Israel. If they had won they would have expelled all Jews. They could have had their own sovereign Nation for 78 years now.

You're not a Zionist, but you then proceed to make a Zionist argument?

The Palestinians were the only people living on the land before the settlers came. That included Jews and Christians, because Palestinians are not a homogeneous group.

The Zionist settlers are not indigenous, they had no right to settle there. They also took the UN resolution and just started a war where they razed 500 villages. I'm sure if the Palestinian side had won, they would have expelled the settlers. But that is only natural. And beside the point, because the Palestinians didn't start the war, and of course uou expel invaders.

"The Palestinians were the only people living on the land before the settlers came"

And native americans were living in Canada, US, and Mexico before settlers came. The Palestinians are hardly unique in losing land in a war. In fact this is basically the norm in human history. What is almost unique is how Palestinians have made their lives much much worse pathetically trying to get the land back when they have absolutely no chance of doing so, anymore than Native Americans do.

The jews living in Israel in 2026 are indigenous in the sense they are NOT leaving so any scenario where they do is a stupid fantasy. the Palestinians DID start the war, they rejected their own sovereign state in favor of gambling for everything and lost. But they have spent 78 years just refusing to accept reality while Israel has grown rich and powerful.

The Palestinians did not start the war. Israel has started every war against the Palestinians, including the 1948 one.

Israelis are not indigenous (there are rare exceptions of Mizrahis or Jerusalem Palestinians with some form of citizenship), they are colonists.

Arabs stared the war when they rejected the UN offer of a sovereign internationally recognized Palastinian state and tried to destroy Israel instead. They lost. In retrospect this was very stupid.

It doesn't matter if Israelis are indigenous or not because 7 million people are not going to leave.

What is plan Dalet? Rejecting the establishment on your land is also not a declaration of war. Israel was the one who started the hostilities

Jews legally migrated to the area and bought land starting 1882 driven by the rise of the Zionist movement and pogroms in Eastern Europe. If the Arabs had won the war they would have forced every last Jew out of the area and Israel would never have existed. When the Jordanian Arab Legion took control of the Jewish Quarter of the Old City of Jerusalem, the ancient synagogues were destroyed, and the entire Jewish population was expelled or taken prisoner. The same happened in the Gush Etzion bloc. If the combined Arab armies had won the war, the Jewish state would have been dismantled, and the Jewish population would have been expelled or killed.They lost but have rather pathetically refused to accept this for 78 years now.

> rare exceptions of Mizrahis

It's not rare at all, the majority of Israeli Jews are Mizrahi.

That's not true, it's not a majority. It is true that Mizrahi are ~40-45% of Jews in the country, but all Mizrahim are not all indigenous to Palestine. I was referring to the subset of Mizrahi who are Palestinian.

What do you mean by “Palestinian”? They’re Palestinian by the classic definition of “those who live in the region of Palestine”.

If you mean citizens of the State of Palestine, that’s a political matter, and zero Jews have that citizenship.

If you mean something along the lines of “unbroken lineage of ancestors who never left Palestine", that would also exclude many people who we all consider Palestinian, such as Arafat himself who was born in Cairo.

By Palestinian Mizrahi, I mean Mizrahi who are culturally Palestinian. Who have some ties to Palestine. Someone you could call native.

Any Jew certainly has cultural ties to Palestine, if we're talking about the region rather than the modern national identity with the same name.

There are more Muslims living happily in Israel than Jews in all Muslim countries combined.

More specifically there are 0 Jews living in areas under control of the Palestinian Authority, or in Lebanon, Syria, Egypt, Jordan, and Yemen.

Illogical collectivist blather suggesting that therefore, Muslims should be treated as a collective, and deprived of rights in Israel? Also, what’s with the “happily”?

No, it just points out how hypocritical Muslims are being.

Hypocrisy is an individual quality, not a collective quality, Zio!

"Zio"

Thank you for providing a perfect example of how the word zionist has become a replacement for kike among jew-haters.

Common term for about 75% of democrats under 50, who must be Jew-haters, of course. Sorry, ethnonationalism just isn’t a popular ideology, and the word is, in practice, just a shorthand for an instance of that. You’ve lapped up hasbarist propaganda to such an extent that it’s started to be reality.

Thank you for providing a perfect example of how the word zionist has become a replacement for kike among jew-haters.

You can stay paranoid or you can choose to face reality. Up to you.

Reality is that irrational hatred of Jews has become normalized among the far left.

The left? The left is against Israeli violence. The far right hates Jews.

Let's compare figures like, say Nick Fuentes and Hasan Piker.

Fuentes regularly spouts vile anti-semitic rhetoric, painting up a picture of Jews as greedy schemers.

Piker always makes it clear he is talking about Israel, not Jews. Painfully so.

I have not seen a single self-odentified leftist say anything anti-semitic IRL. But I have seen right-wingers do this.

And this same story is reflected in politicians statements as well. The right is anti-semitic.

The ADL claims that the left is, but that is because the ADL wants to conflate Israel with the entire ethnic group of Jews. Which is an obvious and silly bad faith trick. You're also doing the same thing.

Hasan Piker is a very bad example because he supports the Houthi, and the Houthi are VERY anti-semitc, their flag literally curses ALL jews

"God is the Greatest, Death to America, Death to Israel, Curse be upon the Jews, Victory to Islam"

Ah, yes. So if Piker, when looking at the Houthi/Israel dichotomy, is more sympathetic to the Houthis, that means he adopts all of their positions?

That doesn't follow. Not even a little bit.

That is just a blatant lie? There are Jews living in all those places. Not that many perhaps, but they exist.

And it's not really about religion. It's about Palestinians specifically, who are indigenous to Palestine, and are under Israeli apartheid.

The Palestinians who have been exiled by Israel, and their children, cannot live where their grandparents lived (even though they should have a right to return, under UN resolutions that Israel has accepted), but any Jew from, let's say Brooklyn, does.

Also, Islam is the only faith in Israel which is not allowed to self-organize. It is singled out among all religious communities as the only one who is not given this right. Which is of course incredibly discriminatory.

There are a lot less Jews living in Muslim countries after Israel was created than before. 800,000 were expelled and mostly moved to Israel. You don't hear them whining for the right to return.

There isn't a single jew living in Gaza. Why is that?

It is very much about religion. Hamas is an explicitly Islamic supremacist organization that calls for the destruction of Israel as a religious obligation.

What do you mean there isn't a single jew living in Gaza? How is that relevant? That is entirely beside the point.

If you had the option to live as a full citizen in Israel, being told you're part of the dominant ethnicity that's treated as human beings; or staying in Gaza, where you are being bombed to death by the IDF, I wonder which you would choose?

Your argument is essentially stating the conditions of apartheid as negative for dominant ethnic group. I didn't know someone could be so divorced from reality.

Between apartheid and genocide Hamas supporters really love using words wrong.

Where's the Hamas supporter?

You sound like one.

Based on what?

Your comments

I'm going to need some more basis than that.

Otherwise this is just the good old "Anyone who criticizes Israel is Hamas" trope.

Very bad news for Gemini - the brief comeback with 2.5 Pro last year looked to be driven by Noam

Don't think it matters in the long run to be honest. The models have no moat, they are becoming a commodity.

Besides that, Google is in a pretty good position, they're not bleeding money on AI like Anthropic/OpenAI, and they own product verticals where they can integrate it. Plus they have a mature ads-model which is what might actually drive a bit of revenue for LLMs.

And they've had some initial success with TPUs which could be a major differentiator in the future.

Yup, and they have the Apple partnership for now as well. Much better position generally than OpenAI in my opinion.

I don't think you're honestly accounting for the engineering behind the progress models are making. If it was just a matter of compute on hand and iterating, Meta would be neck and neck with Ant, OAI, and Google, but clearly you've gotta have more.

Noam has a deep expertise in these systems at every level, both algorithmically and at production scale, and knows how to leverage things at different levels.

It's not like Google won't have anyone else that can do what he does, but at the same time, it's an implicit criticism of Google's culture, operations, development, and overall AI program. Shazeer is well past the point where the paycheck is the deciding factor, although I'm certain he is very well paid. Having the freedom to innovate and build free from the corporate fuckery of Google and Facebook is probably more valuable than the pay raise he got with the move, and OAI has the advantage of not having to cope with decades of corporate cruft and inertia. They'll get there - all corporations do - but they're relatively young enough to still be nimble.

I honestly don't think that matters for multiple reasons:

1. There are already multiple "sota" models on the market that compete with only marginal gains between them (OpenAI, Anthropic, Google/Gemini) and some that are catching up (DeepSeek, Qwen,..).

2. The fact that something is a hard engineering problem does not mean it's generating revenue. So while what you said is true, deep expertise is required to push the industry forward, I don't think that is going to matter for the bottom line of these companies. Hence why I think the models don't give a company any 'moat' in a capitalist economy.

> Noam has a deep expertise in these systems at every level

As do thousands of people say this point. You think the head of deepseek doesn't?

> models have no moat

Possibly true. Any smart innovations developed by one organization will be smuggled into others.

Training, inferring, and data collection, infrastructures are definitely moats. High-volume usage feedback is also hard to come by for new entrants.

And Google has all of those. Custom silicon, more data than anyone else and probably the most comprehensive data collection system, and phones in the hands of 73% of the global smartphone using population to push gemini into to get high volume usage feedback and even more telemetry and data.

yeah, sure, look at anthropic revenue, what is it if not the moat? you can argue for how long but for them good model = the fastest growing company ever.

Revenue is not a metric of success at all.

Grabbing market-share if you have investors that are ready to burn cash infinetely. Find a hot niche, buy a banana 1 USD, sell it for 0.10 USD.

Example: Cursor, they became popular because they were selling ChatGPT unlimited for 20 USD / month.

When they launched, just a reskinned VS Code, "fastest growing AI company"

No coincidence they were bought by SpaceX, who wants to consolidate revenue even if non-sense as long it helps other investors to exit. It shows rapid growth.

Profit is the real moat.

One example: Nvidia. Proprietary tooling, proprietary IP, proprietary hardware, no alternative, expensive.

Revenue is moat. Ask Amazon. Or Alibaba. Or Temu.

You don't know what Cursor's game plan was. Maybe acquisition was their plan.

Buying at $1 and selling for $0.1 is still viable as long as they have money in the bank, until they achieve their goals. Most startups start out that way. Even giving away their services for free.

Obviously there will be failures. Doesn't mean they have no moat. Can you say a business with 100 customers and $1000 debt is less viable than one with a single customer and no debt?

I think the 'models have no moat' thing is overblown. Only like 3-4 companies in the entire world have cutting edge models, that means there is some kind of moat...

money. but it eventually runs out

A little IPO is the solution.

Don't we all want to (automatically) and passively invest in a company losing billions of dollars ?

At least we can diversify our portfolio from SpaceX.

Pre-Quote: "We are all going to lose, hundreds of billions"

Grok and Meta. Both have money and compute, both have shit models. Also Google. Has money, models not so good.

Money.

That's their moat.

Maybe also stolen copyrighted content that cannot be found anywhere else now, so they are the only ones who can train on it.

Meta has tons of that, but no frontier contender. Clearly there’s _something_ more to the equation than money

I think when you follow this stuff every day it's easy to lose perspective of the rate of change and these leads seem more profound than they really are when you zoom out a bit.

I'm no super-insider, I only hear industry scuttlebutt like everyone else, but I have about a 95% confidence that the last 18 months has just been about more and better, without any kind of real leap or breakthrough. More hardware, more data, better technique. Well, technique diffuses as people change companies, hardware can be built, and data can be gathered (or stolen!).

From my admittedly outsider perspective, the only years-long moat there is who has the most hardware. If you have the hardware, you can give away the compute to get the data (hello, subsidized subscriptions!). Technique can simply be hired. The only durable, multi-year advantage is the hardware.

So is that a moat? Sure, but it doesn't have a whole lot to do with the leading model companies of the moment. ASML is the real moat, and so it's ASML China is besieging, correctly (IMO) identifying that everything else can be caught up easily enough.

Check back in a few years...

I feel like the models have no moat paradigm died when a single model expanded past the memory of single GPU slices. The moat is hosting the model. Even paying a server host to run a rack of GPUs has immense upstart cost, and then you're still struggling to compete on the add-ons of the things on top of the model (prompts, validation loops, etc). You can only throw so much money at a problem.

Many different companies host the open source models. Where's the moat there?

I mean you got me there. There will be places who do have the means to build up massive GPU servers. There's just a lot more to it and I don't know if we're going to pinpoint an exact catch-all moat.

Question one: How much did this cost OpenAI?

Question two: Why are OpenAI spending that money taking talent from Google, who can definitely outspend them for talent, and not Anthropic, who are leading the market and are at least somewhat financially constrained.

Reporting on this seems to indicate that people at Anthropic are significantly more loyal, and that attempts to poach by OpenAI and Meta have been largely unsuccessful.

Their options are probably insane sunk cost, hard to steal an engineer who has Xm in potential gains if they choose to stay.

People seem to have turned down offers that would have netted out more upside for them, so it doesn't seem to just be that. Anthropic seems to lure in the true believers, whereas people are highly skeptical of Sam's motivations these days (particularly after how much safety/alignment has been reportedly cut).

But I'm sure for at least some folks, this is true, given recent valuations.

Anthropic is a cult making a god.

Allegedly OpenAI is struggling to jump to bigger models and had serious issues in the past (4.5) and also allegedly Shazeer is just the right guy for that. OpenAI is having issues hiring talent as most SF-style people want to go to Anthropic. Shazeer seems more politically aligned with OpenAI. But it's all speculation.

I'm curious to know the hype behind the hiring for Karpathy and Noam. In the sense that did oai and anthropic do that for sort of long term and potential new directions (investing in them so they come up with the new transformer). Because it definitely cannot be just a regular filling vacant roles.

Because I think as far as running the existing models and handling whatever nuances, it must be well understood by oai and ANT -- but you don't what you don't know.

Sorry to bring up the elephant in the room - but could this decision be in part the opportunity to acquire large amounts of stock before a massively inflated IPO?

Google acquired his company in 2024 for $2.7Bn with him taking about 40% of that. I'm quite sure that no matter where he went, any lab or his own start up, he would be fine financially.

I'm sure he was fine financially when he first worked at Google - without leaving to found the startup as well.

But money at that level isn't about being financially secure - to have a roof over your head and food to eat - it's about power.

Money at that level gives you the ability to shape the world in ways others can only dream of - whether that be starting your own company where you can set the values, funding a cure for Malaria, or political lobbying.

Depends on whether the person in question has strong views and a strong belief that they are in the right.

Full disclaimer - I have no insight or knowledge about this particular person - just making the rather obvious and general case that joining OpenAI now at a senior level is likely to generate a serious windfall, and such a windfall is power.

As I said, no idea what motivates this particular person - don't know them at all - the money may be entirely coincidental and it's all about getting stuff done - but he did choose OpenAI rather than somebody like Anthropic....

BTW I notice John Jumper chose Anthrophic ( or vice a versa ).

AI hiring starting to look like sports free agency.

Karpathy to Anthropic, now Noam to OpenAI.

I thought Karpathy was going to OpenAI?

https://x.com/karpathy/status/2056753169888334312?lang=en

For those who missed: Gemini coding and agentic capabilities have been lagging the sota models (Opus mostly) since Dec 2026. If you're a co-lead and your model is underperforming there has to be some consequences. I don't know as a fact if this has anything to do with Noam's departure, but work performance is never about past successes.

I wonder about the motivation to switch teams. What has Google done wrong? Was he tired? He could retire, open his own lab, raise capital. So many opportunities, why go to OpenAI? Folks talking about the amount of money paid, wasnt he the guy that was acqhired for billions? would OAI pay billions (basically to google) to get him?

At this point it might be about talent.

He could raise and build his own company. But the ability to attract the level of talent that Google/Anthropic/OpenAI have is a different story.

I guess this means Google is nowhere close, to even discern a hint of an AGI? So when Demis Hassabis says AGI...could arrive in just 3 years he has learned the best from Larry Ellison?

I would guess it means Sam Altman gave him more money.

And threw in a sweetener when he seemed hesitant: he can say whatever he likes about trans people in the workplace and not worry about being PC.

Nobody in the space has seen a hint of AGI.

Although I can't fathom why we'd want to? Like what is the advantage of giving tools sentience?

He was one of the leaders and not the leader

And Deepminds Demis Hassabis was the single other? Or were there more?

So didn’t they get on? The latter is in London so time difference to put up with too

Its getting pretty lame that we talk about the these guys like they're football players transferring teams.

Speak for yourself, my Fantasy Developer League is crushing it this season

I feel like there was a scene in Silicon Valley about a developer fantasy league.

Krazam already has a video covering this exact idea.

Fantasy FAANGball

https://www.youtube.com/watch?v=KIZt9YPAPZo

How do I ̷g̷a̷m̷b̷l̷e̷ sports bet on this

It could be the opposite. Those are really useful people, they deserve this more than football players

Idk, football players actually make a bunch of people happy and entertained. 80% of the United States wishes this tech never existed.

What they're working on is just making peoples jobs, skills obsolete and trying to invent machines that will concentrate the worlds wealth into the hands of the people who own those machines.

Very few people interpret football so much that the actual frontier work of the best players matter. Out of 30 friends I know who like football only 1 of them could explain what’s going on in the field technically. For most people, pro players are replaceable.

Popular entertainment and unique progress of human civilization can’t be really compared either

> For most people, pro players are replaceable.

I'd argue that professional sport is the closest thing to a true meritocracy - doesn't matter who your Dad knows - you ability is there for all to see on the pitch.

And at the team level - if cosy cliques form, again - team performance doesn't lie - hard work, team work and talent is ultimately what delivers results on the pitch.

The other interesting part of professional sport is that the 'workers' have managed to capture more of the value than is traditionally the case - this is precisely because they are so hard to replace.

If you think professional footballers earn too much and are interchangable - feel free to try and get in the team.

I only said top scientists and top engineers deserve as much fame / respect / gossip as top football players yeah

Sadly most science and engineering is very capital intensive.

So take this scenario - I'd argue that if you want to make progress in the field of these particular ML models, then you are going to need resources ( compute/data etc ) that is beyond most individuals capability to muster. ie you have to join a company with resources ( or persuade somebody to give you them ).

Right now there is one of those scenarios where capital is chasing talent - and so talent, if they are so inclined, is able to make the most of that.

But in normal times that's typically not the case - most of the time scientists are chasing the capital ( directly or indirectly in the form of a job in a well resourced company ) in order to be able to science, rather than the other way around.

Having the whole world connected to top sports players also costs a lot of money, it doesn’t happen naturally

To become a good scientist you don’t need much classic capital, you need a good environment. And for ML you only need one computer for yourself or you can rent online

There are still big inefficiencies for those who have capital to discover good scientists / engineers. Lots of them are unknown.

But if there are top ones famous it will bring more people to study those fields

> Having the whole world connected to top sports players also costs a lot of money, it doesn’t happen naturally

The existence of pay-per-view sports TV wasn't a pre-requisite for professional football - that existed way before - clubs self funded from gate receipts, local business sponsorship - they grew out of the local communities.

Sure, global TV has brought in the big money, but it wasn't required for the game to exist - but the opposite is true - pay-per-view sports TV is very dependent on sports like football.

> To become a good scientist you don’t need much classic capital, you need a good environment.

Pretty much all scientists learn their craft doing a government funded PhD in government funded labs using government funded equipment. ie governments provide the capital. People simply aren't self taught.

> And for ML you only need one computer for yourself or you can rent online

In theory - but modern AI is so resource intensive, good luck competing with the likes of Google/OpenAI, even Deepseek like that.

No, the way it worked in the past for pro sports wasn’t enough to have those players worldwide famous and gossiped all the time

We need to make science more popular

We disagree on learning science and engineers, this doesn’t require physical capital, it only requires human capital

> No, the way it worked in the past for pro sports wasn’t enough to have those players worldwide famous and gossiped all the time

So? The point is football doesn't require this. It's not necessary for football to happen. The first Fifa world cup was in 1930.

> We disagree on learning science and engineers, this doesn’t require physical capital, it only requires human capital

Try building a bridge without any money. Try detecting the Higgs Boson without CERN. Sure Peter Higgs can come up with the idea of the Higgs Boson with little capital ( though somebody still paid for his living expenses - he wasn't a self funded gentlemen scientist ) - but that's the exception - most of the work is like CERN - and requires significant equipment and capital.

> There are still big inefficiencies for those who have capital to discover good scientists / engineers. Lots of them are unknown.

The way it works in academia is that scientists compete with each other for the limited capital ( grant funding and jobs ). Not the other way around.

> But if there are top ones famous it will bring more people to study those fields

Is the problem lack of talent in these fields or the narrow allocation of capital?

Is it really true that Noam is the only person in the world that could have done what he did or where their in fact lots of people who would have succeed given the same opportunities?

That's not to devalue what they did or the impact - and I'm all for recognising the contributions of scientists to society - but the reality is, for the most part, talent competes for capital rather than the other way around.

I'd also point out that I suspect the high profile appointments of people like Noam and John to OpenAI and Anthropic is as much to do with adding star quality to the IPO as much bringing in talent ( and that's not to diminish their talent ).

You are working around the reality : Noam’s name is good on paper, just like a pro footballer player. His name re assures investors, reduces risks.

This is good.

I don’t care that some are jealous of him because they think they are as good as him in linear algebra.

We're a community of geeks. We admire Tesla, Feynman, Linus and such. For me they are far greater than football players

wait this is kinda brilliant tho

What's the AI equivalent of NIL?

This situation is kind of like backend NIL value. His value to OAI isn't just the work he'll do "on the playing field", it's the perceptual value of "OAI just hired the guy Google paid >$2B to get back" right before their IPO.

Have you seen the Krazam fantasy FAANGball sketch? https://www.youtube.com/watch?v=KIZt9YPAPZo

It's funny, but with the AI hires/moves it feels more like satire now.

This "guy" is worth on the order of all football players put together.

In this case, it's not a new thing ... back in 2005 (yes 21 years ago), people talked about the achievements of Noam Shazeer at Google (and Jeff Dean and Sanjay, etc)

I always appreciated Jeff having a level head ... which this article seems to confirm:

https://www.yahoo.com/news/articles/google-cracks-down-posts...

I wonder if the ideological censorship described in your link is part of why Noam decided to leave.

Shitposting about politics on internal boards is "having a level head"?

I've seen engmisc and industryinfo, and I agree they are sometimes insufferable but having a level head would be ignoring them.

Reread the comment: Shazeer was the one shitposting about politics, not Jeff Dean. Dean called him out over it.

Oh, my bad, I mixed up the names on the comment. Yeah, Dean did always seem to have a level head from what little I've seen.

I think it's more about how the products that impact our lives might change and what might flow down to us becasue of that.

From the excited comments and fanboyism, I have to say KRAZAM predicted the cult of personality that has infected the AI space.

https://www.youtube.com/watch?v=KIZt9YPAPZo&ra=m

Google seems to have difficulties keeping their AI talent.

What is going on at Google?

The engineers let the know-nothing tech illiterate MBAs drive the ship

I hope this doesn't impact Google's progress on open models.

Is Shazeer known to be opposed to open-weight releases?

OpenAI hasn't released open weights since GPT-OSS-20/120B. Google has the Gemma line.

I wouldn't expect OpenAI to start releasing open weight competitive models again, but I could be wrong.

Their models are the only moat they think they have left, which at this point is more of filled-in wet circle of dirt.

Huge blow to Google.

I doubt that the money had anything to do with it.

I also doubt that the state of the technology at OAI vs. Google had much to do with it, Google is behind no doubt, but the gap is not as far as we know, insurmountable.

I suspect that this is a leadership clash. Noam was working in GDM. GDM somehow went away from coding and RSI into "world models" and that has played out very poorly. Who made that call? Who was still playing politics?

Given this is Noam the list of people that could be pissing him off is very small: Demis, Sergey (?!), a couple of VPs in GDM.

What the hell happened?

It's entirely possible he got in a flame war over political issues with other Googlers and Google asked him to leave.

This is what you call a PR hire.

Good luck Noam, Gemini is a great piece of work.

How to be a legend like him?

Shouldn't ai create ai research at this point?

Tell me open ai are in emergency mode without telling me they are in emergency mode

Niceee

Oh! Big deal. Exciting.

Silver lining: given the leaked financials of OpenAI, he might very well be joining a sinking ship.

Also, why didn't they nail him down contractually when they bought character.ai ... isn't that pretty standard with these type of superstar (re)hires?

You can't force someone to keep taking your money (that's indentured servitude), you can only incentivize them to stay with increasing amounts of money. Google almost certainly did do that. Probably by vesting his hiring bonus over 2-3 years.

OpenAI is in a unique position right now to grant pre-IPO options (probably in the form of RSUs). And they wanted him badly enough to grant the extra options necessary to effectively 'buy out' whatever unvested Google bonus he's walking away from.

Yah, I guess Cali doesn't allow non-competes or something like that.

LOL.

Reply on news.ycombinator.com