gwern

> Now that AI is actually making advancements, very little of that work seems to have had much effect. Theres the famous Vonnegut qoute about the combined effort of all preeminent artists protesting the Vietnam war having the effect of a pie dropped from a step-ladder.

One of the interesting things about any topic becoming the 'Current Thing' is that you get to see people making utterly irreconcilable, completely contradictory interpretations of the same public evidence while still somehow reaching the same conclusion.

For example, the day before you commented describing the effect as equal to a pie being dropped on the ground (ie. nil) and that is why they are bad, Palladium published a long (~5.8k words) piece arguing that they had a ton of effect... just in the opposite of the intended direction, and that is why they are bad: https://www.palladiummag.com/2025/01/31/the-failed-strategy-...

Obviously, you can't both be right. (You can both be wrong, though.)

No, it's the other ones that are the problem, the ones you're trying to find. You need to maintain a fixed width of characters, and while you know what characters the secret words tokenize from, and so there's no problem there (you just record the # of characters, turn them into tokens, and you're fine), all of the possible chaff text could have the wrong character size. You sample a possible text of tokens, and the likelihood is good and it reads fluently and the secret words don't stick out, but then the first line has 15 characters too many, the second has 5 too few, and so on, simply because of the vagaries of BPEs.

(And you can't replace the grill positions with relative indexes like 'the ith, mth, and nth tokens' because the tokenizations will all change... Maybe a more complicated relative index like 'whitespace separated character-ranges'... Well, whatever you do, it'll be annoying when it's a BPE LLM instead of a character-level model.)

This would be a perfect fit for a LLM (albeit a bit complicated by byte-pair tokenization and decoder models rather than bidirectional). You can clamp the grille words, and just generate samples with those as requirements until they reach a good likelihood and are highly fluent and indistinguishable from non-message carrying text.

This smells LLM-written (which seems very likely given the rest of this site's content). There's a lot of weird apparatus like defining vocab, a very poor explanation of Kalman Filter which wouldn't help anyone, before jumping into the 'EIF' and then almost instantaneously concluding with takeaways. I'm reading this and not learning anything but buzzwords. And this is +14? Are you guys actually reading this and getting anything out of this?

Source: https://www.theverge.com/news/600542/waymo-test-cities-las-v...

> The problem, as far as I can tell, is that if all this works perfectly, this will be simply the human-programmed Minecraft we already have, except far more expensive to run. The original Minecraft is already infinitely playable, thanks to the way it randomly generates a 7x-larger-than-planet-Earth landscape with each new game. You can't use generative AI like this to get a new kind of game, only a better simulation of the game you trained it on.

Oh come now, that's not true and is showing a lack of imagination, especially from someone who's been covering generative models this long. There's lots of things you can do with an accurate differentiable model of Minecraft - just like there's lots of things you can do with a LLM or a diffusion model beyond just 'generate a random sample'. (Just think about all the things you can do with CLIP steering a diffusion model...) Imagine describing GPT-3 as saying 'as far as I can tell, if this all works perfectly, this will simply be Common Crawl, except far more expensive to read'!

(Tea puns are generally too easy.)

> Also this appears to have been written in 2012. It has nothing to do with modern AI (which I assume is why it was posted today).

Given that it is written by Scott Alexander, it probably was in fact about AI, if not necessarily neural networks per se. (Specifically, about the problems with 'oracle AI' or 'tool AI'.)

Oh, I'm sorry. That should be Mellor 2014, https://pure.manchester.ac.uk/ws/portalfiles/portal/54552919...

(If you are curious, I was playing around with best-arm finding algorithms for taste-testing: https://gwern.net/water )

My apologies to all the D&D sessions which take more than 150k tokens, then.

HN does, but it's still worth pointing it out, so the threads can be merged.

Yes, I've tried out both: ordering it to ask me questions upfront, and sometimes restarting with an edited 'report' and a prototype implementation for a 'clean start'. It feels like it sometimes helps... but I have no numbers or rigorous evidence on that.

A note on the Thompson sampling: classic TS is optimizing for regret, while the goal here is to identify the 'best arm' (ie. the actual secret value at that position - it's like a 0/1 loss). There is no partial credit in getting part of a password correct, it still won't work. So it's a https://en.wikipedia.org/wiki/Multi-armed_bandit#Best_Arm_Id...

You can tweak the Thompson sampling to handle best-arm, like https://www.escholar.manchester.ac.uk/api/datastream?publica... "Ordered-Statistic Thompson Sampling". I am not sure if the pseudocode TS presented by OP winds up doing the same thing.

That is the type of thinking I was criticizing. Your standard swiss-cheese complex-failure model doesn't apply when you are in an adversarial security setting.

When you do a post-mortem on a complex failure, the outcome proves the accusation: "obviously the fire suppression was inadequate, because it didn't suppress the fire; QED". This sort of hindsight is fine for a regular failure, because the steps in the failure happening at all show that they had large probability of happening - if the fire suppression system let the whole ship burn down from a minor accidental fire in the writing room, it was obviously inadequate. (And so on for the other flaws.) But it doesn't apply if you have an adversary, like a psychopath like Rogers. In security, you can have a failure which will happen once in the lifetime of the universe with random events - and which happens 100% of the time when there is an attack.

Perhaps in reality the fire suppression method was well-designed and adequate to the task... except that the attacker used an incendiary with its own oxidizer (the acid+powder) which couldn't be put out. Perhaps the 'poor design' wasn't so poor, and the attacker simply found the weak point. Perhaps the departed captain wasn't so bad, and it was the careful choice of poisoning him at the end of the shift to replace him with an officer near-delirious from sleep, so conked out he's asking if he's dreaming, and who couldn't even order an SOS until well after he should've. Perhaps the lack of fire alarm coverage in that one room wasn't a damning fault, because the attacker would've simply cut the wire before starting the fire if it had had coverage. (A random accidental fire cannot choose to ensure it goes undetected by cutting the alarm wire first. An intelligent attacker can.) And so on.

When you are dealing with a technically sophisticated, patient, psychopathic attacker with zero remorse or concern for human life, who is an insider, and who has been casing the joint for months or years, and who will carefully wait for and arrange the worst possible circumstances, it may simply not be possible to harden the system against such an APT at acceptable cost and performance, particularly when the attacker is willing to put himself at mortal risk for hardly any apparent gain. (Notice that you aren't going on about all of the failures which let Rogers go through his whole life of crime - only about the ship. Why is the ship poorly designed but not society?) Most of society is like this: we cannot ensure zero crime or error or fault. There is no way to stop someone from, say, taking a kitchen knife and going out and stabbing people until the police shoot him, or from renting a truck and driving through a crowd of pedestrians before killing himself.

So, the existence and likely culpability of Rogers completely changes the meaning of the post-mortem claims, and renders them a lot less credible given their naive view of all events as accidental and due to incompetence or bad design, rather than this extreme level of irrational, spiteful, self-destructive evil.

> At least from Wikipedia's description

The description which mentions barely anything about Rogers?

What people are discovering with the latest models is that often their errors are due to entirely reasonable choices and assumptions... which happen to be wrong in your specific case. They call a library you don't have installed, or something like that. Short of inventing either telepathy or spice which can allow LLMs to see the future, it will increasingly be the case that you cannot use the best models efficiently without giving them extensive context. Writing 'reports' where you dump in everything even tangentially relevant is the obvious way to do so, and so I would expect future LLMs to be even more so than o1-preview/pro.

I think OP makes a good case that the cause of the fire was probably the shortsighted poisoner, child rapist, human/dog murderer, embezzler, thief, and bomber who apparently had grudges against the captain who mysteriously died shortly before the fire mysteriously started at the worst time & place possible.

Their creative-writing dataset seems to have been literally written by ChatGPT: https://www.reddit.com/r/LocalLLaMA/comments/1i1a88y/minimax...

Duplicate: https://news.ycombinator.com/item?id=42718166

> I don't think you're going to find an LLM with a large enough context window to have a meaningfully involving story spanning multiple sessions.

First, you don't need much of any context window because you can finetune the LLM. Don't mistake specific engineering choices and tradeoffs and deployment convenience for intrinsic limitations of the technology.

Second, LLMs like Gemini now have context windows of millions of tokens, corresponding to millions of words. Seems like enough for 'multiple sessions'.

> An LLM isn't going to craft a story element tailored to a character, or more importantly, an individual player. It's not going to understand Sam couldn't make last week's session. An LLM also doesn't really understand the game rules and isn't going to be able to adjudicate house rules based on fun factor.

An LLM can do all of that, and you definitely do not know that they can't.

> They'll be no better a game master than a MadLibs book.

They've been better than a Madlibs book since AI Dungeon 1 which was like 6 years ago.

Why is that not just 'transclusion' https://en.wikipedia.org/wiki/Transclusion ?

Except a special-case of transclusion: high latency and compile-time only. As opposed to doing it with JS dynamically; if Pinboard had an API, or even simply doesn't block frames with HTTP headers, you could simply dynamically call from the static site page to the other site, load the relevant HTML, and present it transcluded into the static site page. (This is how Gwern.net does a lot of stuff: everything from the Wikipedia/Github popups to transcluding other pages.)

You are observing "flattened logits" https://arxiv.org/pdf/2303.08774#page=12&org=openai .

The entropy of ChatGPT (as well as all other generative models which have been 'tuned' using RLHF, instruction-tuning, DPO, etc) is so low because it is not predicting "most likely tokens" or doing compression. A LLM like ChatGPT has been turned into an RL agent which seeks to maximize reward by taking the optimal action. It is, ultimately, predicting what will manipulate the imaginary human rater into giving it a high reward.

So the logits aren't telling you anything like 'what is the probability in a random sample of Internet text of the next token', but are closer to a Bellman value function, expressing the model's belief as to what would be the net reward from picking each possible BPE as an 'action' and then continuing to pick the optimal BPE after that (ie. following its policy until the episode terminates). Because there is usually 1 best action, it tries to put the largest value on that action, and assign very small values to the rest (no matter how plausible each of them might be if you were looking at random Internet text). This reduction in entropy is a standard RL effect as agents switch from exploration to exploitation: there is no benefit to taking anything less than the single best action, so you don't want to risk taking any others.

This is also why completions are so boring and Boltzmann temperature stops mattering and more complex sampling strategies like best-of-N don't work so well: the greedy logit-maximizing removes information about interesting alternative strategies, so you wind up with massive redundancy and your net 'likelihood' also no longer tells you anything about the likelihood.

And note that because there is now so much LLM text on the Internet, this feeds back into future LLMs too, which will have flattened logits simply because it is now quite likely that they are predicting outputs from LLMs which had flattened logits. (Plus, of course, data labelers like Scale can fail at quality control and their labelers cheat and just dump in ChatGPT answers to make money.) So you'll observe future 'base' models which have more flattened logits too...

I've wondered if to recover true base model capabilities and get logits that actually meaningful predict or encode 'dark knowledge', rather than optimize for a lowest-common-denominator rater reward, you'll have to start dumping in random Internet text samples to get the model 'out of assistant mode'.

OP is an example of how AI-generated images are usually clutter. Not only do the images not add anything meaningful to the text, and arbitrary parts of the images could be deleted or randomized without affecting the reader's understanding, most of them could be randomly shuffled without anyone noticing. (Which makes them worse then clipart/stockart: if an article swapped the 'hacker hoodie' stockart with the 'neural net brain circuit' stockart, some readers would at least briefly be confused.)

The bank account anchoring bias may be just the base model predicting as best as it can, as OP described earlier. Imagine a random Internet page where you read "There is V in my X. The Y of Z A, in Bs, is _." What do you expect? V, of course! Because why would a person on the Internet ever join these two factoids if they were totally unrelated (as they are in fact in this example)? That would be a gross violation of basic writing and human communication norms (https://en.wikipedia.org/wiki/Cooperative_principle).

> I'll cherry pick just one example from this article. The author in one sentence dismisses the entire field of psychoanalysis.

Properly. The issues with psychoanalysis are extensive, well known, and he has discussed them in the past, like the dodo bird verdict. If you are somehow still a fan of psychoanalysis after everything, then you are not going to be convinced by two sentences of criticism, or even three, and there is no point in trying when that is not the primary topic; dismissing it in 1 sentence is correct. (And this is your slamdunk criticism you lead with?)

The Waymo criticisms are absurd to the point of dishonesty. He criticizes a Waymo for... not pulling out fast enough around a truck, or for human criminals vandalizing them? Oh no, once some Waymos did a weird thing where they honked for a while! And a couple times they got stuck over a few million miles! This is an amazingly lame waste of space, and the fact that he does his best to only talk about Tesla instead of Waymo emphasizes how weak his arguments are, particularly in comparison to his earliest predictions. (Obviously only the best self-driving car matters to whether self-driving cars have been created.)

"Nothing ever happens"... until it does, and it seems Brooks's prediction roundups can now be conveniently replaced with a little rock on it with "nothing in AI ever works" written on it without anything of value being lost.

Hm... I was definitely under the impression that it is generating SVGs natively, and that was consistent with its output and its recent upgrades like good text rendering, and I'm fairly sure I've said as much to the CEO and not been corrected... But I don't offhand recollect a specific reference where they say unambiguously that it's a SVG generator rather than vectorizer(raster), so maybe I'm wrong about that.

No, I actually was referring to their native vector AI image generator, not their vectorizer - although the vectorizer was better than any other we found, and that's why we were using it to convert the Midjourney PNG dropcaps into SVGs

(The editing quality of the vectorized ones were not great, but it is hard to see how they could be good given their raster-style appearance. I can't speak to the editing quality of the native-generated ones, either in the old obsolete Recraft models or the newer ones, because the old ones were too ugly to want to use, and I haven't done much with the new one yet.)

Have you looked at https://www.recraft.ai/ recently? The image quality of their vector outputs seems to have gotten quite good, although you obviously still wouldn't want to try to generate densely textured or photographic-like images like Midjourney excels at. (For https://gwern.net/dropcap last year or before, we had to settle for Midjourney and create a somewhat convoluted workflow through Recraft; but if I were making dropcaps now, I think the latest Recraft model would probably suffice.)

I am flattered to hear all that.

I think Said is a lot closer to the '1px is broken' than me, but also that it's not really that important. How many websites have dropcaps at all? They are fun to have, but very optional. And so if you find the maintenance to be a burden, you can simply not do that. And then the core CSS/JS basically never breaks. (All of your problems will come from churn elsewhere, like SSL certificates - a major source of feature creep is the demand that everything be HTTPS now - or OS upgrades or SaaS bitcreep.) So my point is that a simple website is as close to zero-maintenance as makes no difference. It's much more complex than plain text files, but that complexity is widely implemented and is stable.

I too am biased towards desktop and have to force myself to make any use of the mobile version, and check for regressions or friction. I find mobile to be incredibly stultifying for any kind of complex work or thinking. Nevertheless! There are a lot of mobile readers (including many people I respect and would like to keep as readers). So, mobile's gotta be good. At least I think we have managed to create about as realistically good a reading experience on mobile as feasible, and serve as a design showcase there too, to shame everyone else.

That sounds plausible, but I haven't attempted to link visitors across devices. I am not sure if Google Analytics even allows that. (The new one is so confusing and hard to use I've largely stopped looking at traffic statistics.) I think it might be hard to interpret such an interaction, though, because mobile users bounce so fast and clearly so distracted & thoughtless & crippled (eg. because they are killing time on a bus), while desktop readers are much more mindful and 'high-quality time', so they are different in many ways beyond what version of the website they see.

Do very many people actually want to claim to show off the latest web dev gimmick? I'd say not. Almost no one is doing that. Even web devs often have quite plain personal websites or blogs which are not trying to show off their mastery of the latest Nodebuzz.js (best viewed in Chrome 234) feature. When people write about their new Proton static site which uses Netlify to cross-compile from Github pages through AWS Lambdafront functions called on demand with a Kafka load balancer to prerender pages for the wasm.js shim and it only costs $100/month after optimization to host their blog with 50 pages and 50 views/day (aside from that time they accidentally created 10MB of logs and it cost $1,000 but don't worry, AWS was willing to forgive it), I am pretty sure they are not going through that to impress an employer! They are going through that because it's a lot of gizmos and puzzles to play with and 'number go up', and the end-result is irrelevant. They are doing it to watch the gears of the new toy go round, as Norbert Wiener said of the atom bomb makers. Which is fine, as long as you're just getting experience (and also not making a new atomic bomb), but may have side-effects in naive people thinking that any of that is a good idea, much less necessary.

Managers & designers, on the other hand, do need churn to justify their existence. Look at the hilarious trends of companies commissioning new fonts where you can't see the difference with some Helvetica-like even overlaying the fonts, accompanied by thousand-word manifestos rhapsodizing over the design process inspired by the cliffs of California and the sensual humanism of Bauhaus (with a cheeky node to Swiss Modernism, and a dangerously daring serif on the 'g').

Oh, I think I would disagree there. As I was saying, browsers are actually amazingly good at backwards compatibility (perhaps too good). Which makes it easy to decide what browser features to use: if it's Chrome-only, sure, it may be gone in 5 years, but you don't have to be a genius to know "maybe I shouldn't make any major use of a Chrome feature that no other browser wants to support, and I should wait and see". I may have mentioned that we have a rule of thumb: we can use anything with >95% global support on CanIUse.com, and anything else must be backwards compatible/polyfilled or not used at all. We've hit a handful of things where the browsers betrayed us (I'm lookg at you, `<srcset>` and `ping` attribute), which you can find on https://gwern.net/design-graveyard but you know, even mid-1990s `<blink>`, which was never standardized to begin with, worked until 2013! So I'll put it this way: I don't think you can name 3 examples of reasonably widespread features, which were standardized and had >95% CanIUse.com support as of 5 years ago (2020), which are now either broken or rapidly being removed and have fallen <95%. (The only examples I can find are still exotic: https://en.wikipedia.org/wiki/Cache_manifest_in_HTML5 , browser FTP support, and https://en.wikipedia.org/wiki/HTTP/2_Server_Push 2 of them would have been bad ideas to use in new code in almost any circumstance, and I suppose for personal websites you could've in theory used the cache for an offline browseable website... maybe. I doubt many people ever did.)

I think there's some degree of anxiety around perfectionism, but also around the aversion of writing (some people find any kind of writing to be very difficult, even when they have just written an essay by talking to you in a chat client), and the fear of pseudonyms being broken or being canceled. (Zoomers & Alphas seem particularly terrified of 'the dark forest'.)

Oh, I assumed the image gen at least was running on the fancy remote GPU server.

But it being on-device doesn't prove it's not a DALL-E 3. Just like GPT-3 & GPT-4 & Llamas come in many sizes, some of which are (probably) feasible to run on-device...

Still, I suppose that probably does imply that it's some Apple half-hearted image-gen model (trained on a lot of DALL-E 3 contaminated data?), since I'm not sure OA would want to risk leaking even years from now a tiny, terrible, hyper-optimized-for-size DALL-E 3's weights.