I never thought I'd see the day that LessWrong would be accused of being biased against near-term AGI forecasts (and for none of the 5 replies to question this description either). But here we are. Indeed do many things come to pass.
35,337 karma · joined April 8, 2009
About me: https://gwern.net/me
What I've written recently: https://gwern.net/changelog
> Reached for comment, LastPass said it has seen no definitive proof — from federal investigators or others — that the cyberheists in question were linked to the LastPass breaches. “Since we initially disclosed this incident back in 2022, LastPass has worked in close cooperation with multiple representatives from law enforcement,” LastPass said in a written statement. “To date, our law enforcement partners have not made us aware of any conclusive evidence that connects any crypto thefts to our incident. In the meantime, we have been investing heavily in enhancing our security measures and will continue to do so.”
So, there's slightly less than conclusive evidence, and how much evidence should you need anyway, given the alternative hypotheses?
Good example of how the bottlenecks are often not where you think they are, and why you have to profile & measure (which I assume Viti95 did in order to find that speedup so early on). The status bar percentage?! Maybe there's something about the Doom arch which makes that relatively obvious to experts, but I certainly would've never guessed that was a bottleneck a priori.
The line " A special shout to our limo service San Diego for that one! " is a spammer insertion in https://web.archive.org/web/20170430095756/http://www.leadho... which makes no sense, and indeed, if you jump back over the gap of a few months to https://web.archive.org/web/20170121041802/http://www.leadho... , it was the much more logical in context " For collectors it doesn’t take long to fill out your collection of them with..." preface.
I wonder if it went offline because they couldn't deal with the security and maintenance?
That's also not true. A base LLM like GPT-3, without any (specialized) training data, alignment, or prompt engineering, embodied quite a bit of agency (able to answer questions, simulate coding, or take actions in a while-loop), and in principle, can be AGI if scaled up. Consider Gato for an example: no alignment, no prompt engineering, just a GPT on data. This is unlike a stone soup which is completely un-nutritious without the villagers donating all the edible ingredients to make an actual soup, where the stone itself does nothing (a LLM definitely does something); the two scenarios are different qualitatively, not just quantitatively. Gopnik's analogy and goal for her analogy are both irretrievably flawed.
> All analogies melt if you push them loudly enough.
This analogy didn't melt after being pushed too far. It caught on fire spontaneously in the garage while no one was looking before half the parts arrived or had been unpacked.
A stone soup does nothing by itself. It just sits there. LLMs do not just sit there: Claude and o1/o3 and r1 are increasingly active agents. Gopnik just plain ignores this, and indeed, says already obviously false things like "For some time, I’ve argued that a common conception of AI is misguided. This is the idea that AI systems like large language and vision models are individual intelligent agents, analogous to human agents. Instead, I’ve argued that these models are “cultural technologies” like writing, print, pictures, libraries, internet search engines, and Wikipedia." But last I checked, I couldn't ask 'a picture' to go research anime for me and summarize the results, nor could I put it in a self-improving loop to learn how to solve advanced math problems I can't even understand.
Ingredients in soups are used up and destroyed, and can only contribute to one soup; copies of text do none of that.
The villagers had to go out of their way to proactively add ingredients to the soup - indeed, that is the entire point of the original moral! OP seems to think that people donated all the data that they had "stashed away on the Internet" (what a phrase). Whereas with generative models, they don't, and in fact, that's a big reason many people are so mad.
Each ingredient added to the stone soup makes up a meaningful percentage of the output; any single contribution to generative models at the billions-scale is usually invisible and that contribution can be omitted without any measurable change on just about any metric.
Ingredients in soup may be tastier, but they are generally not meaningfully more nutritious. A potato cooked on its own is as nutritious and full of calories as it would be in the soup.
Soups can only be eaten, and eaten once. LLMs do a lot of things, which we are still discovering, and are reusable indefinitely, and are already spreading out into fields no one dreamed of (like in psychology & economics, increasingly more research is doing 'in silico' with LLMs as humans).
The villagers also donated their ingredients for free. The stone itself does nothing and the 'soup maker' likewise does barely nothing and contributes no ingredients but the stone. LLM trainers spend literally tens to hundreds of billions of dollars, including billions spent collectively on creating data through Scale etc. (Rumor has it that OA and Anthropic alone are spending hundreds of millions of dollars on expert programmers and other PhD specialties and that this is part of why their models are so much better.) Notably, the little mention of 'Kenya' implies they do it for free as they "jump at the chance" - obviously, the actual Kenyan villagers are very interested in being paid.
Making the soup doesn't make making future soups cheaper nor does it make the future soups tastier; making LLMs drives experience curves which are some of the fastest ever documented, which is why the cost of high-quality outputs has dropped by multiple orders of magnitude in just years, outpacing Moore's law.
So... no. There is pretty much no way in which LLM is like a 'stone soup', except in the vague sense that both involve a lot of humans at some point, I guess.
He's a regime supporter who brings in desperately needed foreign currency to help pay for the nuke & missile program and elite luxuries. Hence, given that he has talked about and posted this, he is almost certainly correct when he says
> Some media has portrayed the idea that North Koreans are embarrassed by the empty hotel or the length of time it took to finish, Cockerell said, "but I never found that to be true"..."That's ludicrous, and never been true. Absolute nonsense. It used to appear on the front of books and magazines, even when it was an incomplete concrete shell with a crane up top. So that's complete rubbish. It was not a cause for embarrassment." Cockerell said the Kim family simply put the blame on America, falsely explaining to the North Korean people that the delays were "the fault of jealous conspiracies" from outside Western powers...Cockerell has also observed how Pyongyang locals interact with the structure. "It's not like people sit on their balconies, watching the slogans go by. It's just part of the nightlife. It's a futuristic building to North Koreans, very modern, unlike anything else."
That is, this post is sort of a quasi 'submarine' for North Korean tourism. 'Come see our unique pyramid, which we think is cool rather than a symptom of totalitarian dictatorship (and pay us a lot of renminbi/dollars/yen)!'
It's because it reads mostly as a retread of the far-better known Samuel Butler pieces like _Erewhon_ years before (https://en.wikipedia.org/wiki/Darwin_among_the_Machines was 1863), and which Eliot likely read given her lifestyle & how closely she echoes him. Surprising to see no mention of Butler at all in OP, given what critical context it is - she was not writing in a vacuum.
Some earlier discussion: https://www.lesswrong.com/posts/goANJNFBZrgE9PFNk/shadows-of...
You didn't show causality, though. You never randomized anything. His study and your observation was purely observational. At no point did you open the hood, get ready to wrap the wires, and flip a coin to decide whether to wrap the wires or do a placebo wrapping somewhere else.
Had you done that, you would have found, per your ultimate explanation, that the wrapping made no causal difference: you did the procedure, and either way, the car turned on. Hence, there is no causality for you.
Mode-collapse. One reason that the tuned (or tuning-contaminated models) are bad for creative writing: every protagonist and place seems to be named the same thing.
> Today, we’re releasing an experimental version of Gemini 2.0 Pro that responds to that feedback. It has the strongest coding performance and ability to handle complex prompts, with better understanding and reasoning of world knowledge, than any model we’ve released so far. It comes with our largest context window at 2 million tokens, which enables it to comprehensively analyze and understand vast amounts of information, as well as the ability to call tools like Google Search and code execution.
There are many strange altered states of consciousness found by experimentation, and I can totally believe that some form of auto-hypnosis + sleep deprivation - which would parallel many well-attested mind-altering practices in many different cultures & religions - could have bizarre effects or induce psychosis and hallucinating demonic entities, and turn an activist vegan into a murderer. (Their specific interpretation of 'unihemispheric' sleep may or may not be true. I would say that it's worth checking... except I'm not sure how one could either prudently or ethically investigate it, given the apparent consequences.)
And it would answer the question many people have been asking about how these unlikely-seeming murderers were made, in a way that the rest of it all does not.
Classic RL reward-hacking! You'll often see 'suicidal' agents when the reward-shaping goes wrong, or they converge prematurely after inadequate exploration. (Tom7's Mario agent famously pauses the game to avoid losing.)
> I’m not sure if there are good ways around this. Rigorous testing, maybe, but it’s hard to improve your algorithm by observation once it’s not making obvious mistakes. This is, perhaps, one of the promises of using reinforcement learning without human play training: if your algorithm is able to achieve human-level performance, it’s probably correct enough to continue well into superhuman performance.
In this case, because the random number generator's decisions are unaffected by your actions, you can use the clairvoyant oracle trick, I think. Just record games, and then do your DFS planning over the entire game with the known sequence of blocks; this then represents perfect play and is a benchmark for your actually feasible algorithms.
A COMPUTER CAN NEVER BE HELD ACCOUNTABLE
THEREFORE WE MUST NEVER DENY THAT
COMPUTERS CAN MAKE DECISIONSWell. I guess he has a good reason to not update it.
† inspired by http://www.rdrop.com/~half/Creations/Writings/Web.patterns/f... , we've been adding 'modified' icons to new URLs. Aside from that, the big Web 1.0 UI feature I am missing is any good equivalent to 'breadcrumbs': https://en.wikipedia.org/wiki/Breadcrumb_navigation http://www.rdrop.com/~half/Creations/Writings/Web.patterns/v... They work in the website's own use of it, because it has a strict categorical design where there is a 'book' nested in a domain etc, and the examples given obey categories. But I've struggled to come up with a good equivalent for Gwern.net, to help the reader visualize 'where' they are. Tags, for all their virtues, which are why they have largely superseded categories, do not give you any good spatial analogy. (There is an argument for breadcrumbs within pages, however, and we've been thinking about a kind of 'browsing history' section which shows you each link or snippet of text you interacted with in chronological order, which might help.) Some of them are better off forgotten, though. I think it's become clear that http://www.rdrop.com/~half/Creations/Writings/Web.patterns/c... weren't good ideas.
What might this... do... for me? Don't dump a bunch of JS which is how I would 'do' whatever it does. What does it do? Like, can I dump the URL 'https://pmc.ncbi.nlm.nih.gov/articles/PMC4543385/' into it and get out nice usable clean text of the abstract, say? What about a complicated PDF like https://gwern.net/doc/psychiatry/anxiety/2025-he.pdf (these are the last two references I added)? What do I get? Do I have to install the whole darn thing just to see what it does?
As it happens, Zuckerberg appears to have concluded that FB needs to still be in the foundation model business: https://www.facebook.com/4/posts/10116307989894321/ Possibly because LLMs are far from 'done', and it's unclear if DS does have it handled indefinitely.