819 karma · joined March 28, 2014
Multimodal models
Language models for art
The Trillions of parameters claim is about the pretraining.
It’s most efficient in pre training to train the biggest models possible. You get sample efficiency increase for each parameter increase.
However those models end up very sparse and incredibly distillable.
And it’s way too expensive and slow to serve models that size so they are distilled down a lot.
Also can you run batchwise effectively like vllm on cuda?
Enough to run multiple agents at the same time with throughput?
What actual benefits do they get?
I guess they can have their own models run faster than the competition on their hardware? But they don't even really have anything that consumers use on the ANE as far as I can tell and local LLMs are taking off on macs and could really benefit from this
Or just try to use them. They don’t generalize as well.
They are benchmaxxed.
It’s petulant the way the EU is throwing a hissy fit after we’ve had lop-sided trade deals for years and funding the entire NATO alliance ourselves.
They act like we’re going to war with them when we’re asking for parity and for their self reliance to increase.
They’re actually very good at speed optimization and can iterate very quickly taking notes on trials and failures and benchmarks. I’ve had it write 10 different attempts in around an hour and benchmark them all then merge and beat very strong baselines in torch
I think from training it's still biased towards simple tooling.
But also, there is real power to simple tools, a small set of general purpose tools beats a bunch of narrow specific use case tools. It's easier for humans to use high level tools, but for LLM's they can instantly compose the low level tools for their use case and learn to generalize, it's like writing insane perl one liners is second nature for them compared to us.
If you watch the tool calls you'll see they write a ton of one off small python programs to test, validate explore, etc...
If you think about it any time you use a tool there is probably a 20 line python program that is more fit to your use case, it's just that it would take you too long to write it, but for an LLM that's 0.5 seconds
Look at Nvidia nemotron series. They hav become a leading open source training lab themselves and they’re releasing the best training data, training tooling, and models at this point.
We need a well managed set of immigration polices or country WILL take advantage of US. These are our military rivals and we sell our most advanced math, physics and engineering seats to the highest bidder. It’s a self districting disaster and it’s not just on us to treat people better.
Look at the rate of Indian asylum seekers in Canada to see the most extreme case. It happens anywhere you extend naivety and boundless good will.
Thats especially encouraging to me because those are all about generalization.
5 and 5.1 both felt overfit and would break down and be stubborn when you got them outside their lane. As opposed to Opus 4.5 which is lovely at self correcting.
It’s one of those things you really feel in the model rather than whether it can tackle a harder problem or not, but rather can I go back and forth with this thing learning and correcting together.
This whole releases is insanely optimistic for me. If they can push this much improvement WITHOUT the new huge data centers and without a new scaled base model. Thats incredibly encouraging for what comes next.
Remember the next big data center are 20-30x the chip count and 6-8x the efficiency on the new chip.
I expect they can saturate the benchmarks WITHOUT and novel research and algorithmic gains. But at this point it’s clear they’re capable of pushing research qualitatively as well.
They charge higher costs than OpenAI and have faster growing API demand. They have great margins compared to the rest of the industry on inference.
Sure the revenue growth could stop but it hasn’t and there is no reason to think it will.
They could make more money keeping control of the company and have control.
The would have taken some time to calculate the efficiency gains of pretraining vs RL. Resumed the GPT-4.5 for whatever budget made sense and then spent the rest on RL.
Sure they chose to not serve the large base models anymore for cost reasons.
But I’d guess Google is doing the same. Gemini 2.5 samples very fast and seems way to small to be their base pre train. The efficiency gains in pertaining scale with model scale so it makes sense to train the largest model possible. But then the models end up super sparse and oversized and make little sense to serve in inference without distillation.
In RL the efficiency is very different because you have to inference sample the model to draw online samples. So small models start to make more sense to scale.
Big model => distill => RL
Makes the most theoretical sense for training now days for efficient spending.
So they already did train a big model 4.5. Not using it would have been absurd and they have a known recipe they could return scaling on if the returns were justified.
So it’s more about the mask modeling objective than Diffusion.
After a long stretch of I just ask codex to: Audit the code base and event changes and describe the data model and all related and possibly overlapping functions.
Plan a new redesign that is simpler, has less code, more reuse and cleaner design,
Execute the refactor Review the code and asses new code and reaudit
… repeat
You can queue these up in codex and it will just go about its way reducing your tech debt way faster than an engineer could.
I think I was the first to write an LLM language and first to use LLMs to write a language with this project. (Right at ChatGPT launch, gpt-3.5 https://github.com/nbardy/SynesthesiaLisp
There is for a fact teams creating puzzles to RL against as training environments. As it’s beneficial to RL training and in particular compute efficient if you schedule the environment difficulty throughout training. There was a great recent paper on this. Creating environment data that generalizes outside the environment is a challenging engineering task and super valuable whether it looks like AGC AGI or not.
Also ARC AGI is general enough that if you create similar data you’re just creating generic visual puzzle data. Should all visual puzzle data be off limits ?
And sure maybe not 2mil of it is usable, but they're reliably pushing the frontier here.
Look at VLLM. It's the top open source version of this.
But the idea is you can service 5000 or so people in parallel.
You get about 1.5-2x slowdown on per token speed per user, but you get 2000x-3000x throughput on the server.
The main insight is that memory bandwidth is the main bottleneck so if you batch requests and use a clever KV cache along with the batching you can drastically increase parallel throughput.
It provides supply in the market and lowers their bargaining power, so the parity in wages is parity in a lower baseline
Also in practice there are salary ranges and the H1B's usually take the lower of the range AND work longer hours at the fear of visa loss.
I built a version of this a few years ago as a LISP
Should be quiet easy if you have some o4-mini results sitting around.
You can be over specified in your prompts and say exactly what types and algorithms you want if you’re opinionated.
I often write giant page long specs to get exactly the code I want.
It’s only 2x as fast as coding, but thinking in English is way better than coding.
You really just want to split those things out of the UI component and keep minimal code in the component file. But having it be code free is too much for real client side apps.
Modern apps have very interactive UI that needs code. Things like animation management(stop/start/cancel) etc… have subtle interactions. Canvas zoom and pan, canvas render layouts that relate to UI, etc… lots of timing and control code that is app state dependent and transient and belongs in components.
I apply the simple principle and move any app state management and business logic out of the component UI code. But still lean into logical code in UI.