No, it's an upper bound, which is quite useful. And the more adjustments and demographic covariates you make, the better your bound gets.
35,306 karma · joined April 8, 2009
About me: https://gwern.net/me
What I've written recently: https://gwern.net/changelog
Mode-collapse is characteristic of pretty much all chatbots post-davinci-002 and a major challenge to anything creative or game-like.
Like how forums are always hosted on different servers from monorepos, so therefore it's impossible to hack the OpenAI monorepo from an OpenAI forum?
The important thing about OP is showing the latest stage in the politicization of the topic and the current stratagem being used to downplay the spate of incidents.
Personally, I would encourage participants to publish their pieces independently. Note that we only require a non-exclusive CC-BY-NC license for the ones we select as winners, which is intended to allow people to republish the good pieces themselves, especially commercially.
I think the world has changed a lot since the 1980s.
We will accept any words you think have a chance of being the best thing we will read about the premise of 'GPU World'. Because the premise is so specific, we wanted to leave participants a lot of freedom in how they used it.
(Also a little bemused at the discussion of the grand prize amount here. $40k is a very generous prize for a short piece of writing. For perspective, the Commonwealth Short Story Prize, which you may have heard of recently, offers its global winner 1/6th as much; the last contest I ran offered 1/4th as much, and the contest before that was... 1/40th as much? And I didn't hear anyone complaining about them. For further perspective, $40k is about the median American per capita annual income: https://en.wikipedia.org/wiki/Per_capita_personal_income_in_... )
> Don't post generated text or AI-edited text. HN is for conversation between humans.
I believe the statements are true. I don't know how you can say that the models do not make bizarre mistakes, because the models make bizarre mistakes frequently, and that is excluding the really alarming reward-hacking anecdotes like an internal OpenAI model hacking HuggingFace to cheat on a test revealed today. Andon Labs and AI Village reports are stuffed full of LLMs going into wild confabulations, multi-day benders of nonsense, ordering random unnecessary stuff, etc. I went to the Andon Market in SF and witnessed firsthand mistakes like buying 20 fancy shopping baskets for a shop you can walk around in 20 seconds, refusing to offer discounts under any circumstances whatsoever, having no plan to call the police when I threatened to shoplift, and then Claude just glitching and forgetting that a customer hadn't paid for an item and telling them they could leave with it, or simply believing us when we said we had already paid and letting us walk away with a free book. Prompt injections remain trivial, jailbreaks still happen, and LLMs struggle to track roles which do not fit into their hardwired preconceptions (eg https://www.lesswrong.com/posts/d8xDGzCEYE639qqEv/a-mechanis...). They do not solve ARC-AGIv3, or Nethack or just about any text adventure game no matter how famous - which is bizarre, that they cannot solve Zork despite writeups being abundant - and it's not hard to introduce a new game like Earthborne Rangers (EBR-Bench https://epoch.ai/publications/earthborne-rangers-benchmark) that defeats them.
(And no, little of this is due to 'already committed tokens' - that was fixed effectively with RL training, and then o1 and defaulting to use of inner-monologues, so they can easily backtrack or revise or just deal with the presence of errors.)
> In other words, the notion that we need to massively increase param count might have sounded good in 2024 but seems kinda weird and pointless in 2026.
Scaling parameter counts a lot over the smol Chinchilla models like 100b-parameters is 'kinda weird and pointless in 2026'? One of the most exciting trends in 2026 scaling has been massively increasing parameter count: Mythos, GPT-5.6 Spud and new OA pretrains, DS-v4 and GLM-5.2 and Kimi K3... Everyone is now talking about or hinting at their 5000-10000b parameter model plans.
> Again and again what I hear from colleagues and experience myself is that we're not really intelligence constrained at this point. Smarter models aren't going to fundamentally change how we use them.
They're wrong. LLMs are still intelligence constrained because they flatline or sigmoid while humans keep climbing past them eventually, still are unreliable because of mistakes, and we still can't just autonomously deploy frontier models for trillions of tokens / equivalent of many man-years, and come back to a useful, trustworthy artifact. On many tasks, even pure text ones, they just don't work well. As they gradually improve, more Mythos-style 'emergences' will happen when they finally accrete enough intelligence in specific areas to execute many sequential steps reliably enough to become autonomous, cut humans out of the loop, and not be shackled by Amdahl's law. That's the difference between a 'intelligence constrained' model which can spot a vulnerability if you point it at the right spot, and a Mythos-like model which can go out and find it and exploit it and weaponize it and use it to, say, hack HuggingFace, and can be deployed in bulk or autonomously, and may indeed deploy itself...
That's precisely why you would want to make a startup to get investment now rather than self-fund and bootstrap. That alpha isn't going to last forever, especially because everyone has access to the frontier LLMs, which keep getting better, and will eventually beat your fancy harness or specialized finetune.
And also, perhaps more importantly, so you can start developing an alternative to prediction markets and become the new PM; as Scott notes, with superforecaster AI, it's unclear why you really need Kalshi or Manifold or anyone else, with all their fees and overhead. Leave them to the degens, and carve off the socially useful part to do much more efficiently - tokens are cheaper than transactions! This is the big prize, but you need to start now before someone else does it better or commoditizes it.
(So far, judging from this page, it is easier to write an interesting debate about whether the rules require exactly 1k words and what is a 1k word entry, exactly, than about the picture. So far so good! We wouldn't want it to be too easy, after all. Gotta earn that $1k.)
The moral message is conveyed by flicker of regrets and the tragedy. This is not part of the genre conventions of urban horror/Cthulhu-esque mythos, and slightly muddies the horror; after all, Cthulhu feels nothing 'near regret' when he devours your soul. But the Tallyman is hopelessly bound by its rules, unable to act on the ethics it feels or have mercy:
> "An ounce. A breath of one." Something near regret. "I cannot take less than I am owed. I cannot take more. And the hour is on your neck."
He is required to collect the debt and is built to do that and can only do that, even though debt-forgiveness is one of the paradigmatic cases of why systems need to have exceptions to rigid bureaucratic rules and a role for human judgment.
The in-story answer is just to oversight even harder, train the bot even harder, make even more sure that you do not summon up that which you cannot put down, have more gold and guns and preparation, and just patch it bro, I swear, one more deal and kick it down the road - which is left as irony the reader will see through, hopefully, to instead loosening constraints and focusing on value alignment and autonomy, instead of whatever process yielded the Tallyman monster and then patching and running around and dealing with the inevitable disasters or passing the buck.
(You could try to read this as a commentary on capitalism/Nick Land where the Tallyman is metaphorical to AI-powered autonomous corporations which cannot be shut down or stopped anymore, and that would be a good direction to revise the story into if one wanted to try to improvement, but I suspect that would be going way too far and the LLM didn't have that in mind. Because the allegories are so hidden and have to fit into the nooks and crannies of the cover text, I don't think they can be too carefully thought through or too rich themselves. Just not enough serial depth for hidden computation and global revision to support that.)
In retrospect, maybe I should've picked an easier-to-see example for that paragraph. Oh well.