Did you publish this anywhere? Would love to read more.
611 karma · joined May 4, 2025
Into world models, reinforcement learning, evolutionary methods and fast ML code
Hopefully they get Mojo to a good place for more general ML, but at the moment it still feels quite limited - they've actually deprecated some of the nice builtins they had for Tensors etc... For now I'll stick with JAX and check in periodically, fingers crossed.
The world model is basically intended as a more true-to-life simulator.
His parents weren't particularly wealthy. More likely: he is exceptionally intelligent, hardworking, visionary, and grew up in an environment which fostered those attributes in a precocious child.
Separately, I think Anthropic are probably the least likely of the big 3 to release a model that uses latent-space reasoning, because it's a clear step down in the ability to audit CoT. There has even been some discussion that they accidentally "exposed" the Mythos CoT to RL [0] - I don't see how you would apply a reward function to latent space reasoning tokens.
[0]: https://www.lesswrong.com/posts/K8FxfK9GmJfiAhgcT/anthropic-...
I work with a good editor from a respected political outlet. I've tried hard to get current models to match his style: filling the context with previous stories, classic style guides and endless references to Strunk & White. The LLM always ends up writing something filtered through tropes, so I inevitably have to edit quite heavily, before my editor takes another pass.
It feels like LLMs have a layperson's view of writing and editing. They believe it's about tweaking sentence structure or switching in a synonym, rather than thinking hard about what you want to say, and what is worth saying.
I also don't think LLMs' writing capabilities have improved much over the last year or so, whereas coding has come on leaps and bounds. Given that good writing is a matter of taste which is beyond the direct expertise of most AI researchers (unlike coding), I doubt they'll improve much in the near future.
I guess at a certain point you're getting at a more fundamental question about the value of AI (plus technology and everything else) - what level of environmental tradeoff is acceptable? One thing I slightly lament about the discourse is that tradeoff is widely discussed in the case of AI, but not in the context of stuff we do. I suspect most people aren't aware that the water use associated with eating a burger dwarves a year of ChatGPT, that a long-haul flight wipes out the emissions savings of a couple years' veganism, or that renewables have their own impacts, like the demolition of Chile for copper.
I think your points around parallelisation and the flexibility of quadratic attention are spot-on though.
If anything, a better look at the economics is a reason to look forward to one of them IPO-ing. I suspect the labs probably could cut R&D and turn a profit, but that might only work for one generation, until they get superseded by the competition.