gwern

This is the epitome of patching symptoms rather than treating the disease. Even if you suppress the obvious syntactic slop like 'it's not X but Y', you have no reason to believe you've fixed mode-collapse on higher more important levels like semantics and creativity. (For example, Claude LLMs have always struck me as mode-collapsed on a semantic level: they don't have the blatant verbal tics of 4o but somehow they still 'go in circles'.) Which will potentially severely hinder the truly high-value applications of LLMs to creative applications like frontier research. To the extent that this succeeds in hiding the brain damage in contemporary LLMs, it arguably is a cure worse than the disease.

You're correct, but when the worst the ChatGPTisms get is turns of phrases like "LeetCode youth finally paid off: turns out all those "rebalance a binary search tree" problems were preparing me for salami, not FAANG interviews." or "Designing software for things that rot means optimising for variance, memory, and timing–not perfection. It turns out the hardest part of software isn't keeping things alive. It's knowing when to let them age.", then I'm inclined to forgive it compared to how many far more egregious offenders are at the top of HN these days. This is a rather mild use of ChatGPT for copyediting, and at least I feel like I can trust OP to factcheck everything and not put in any confabulations.

My point is more that Schmidhuber is saying that the gates or the initialization are the innovation solely because they produce well-behaved gradients, which is why Hochreiter's 1991 thing is where he starts and nothing before that counts. But it's not clear to me why we should define it like that when you can solve the gradient misbehavior other ways, which is why https://gwern.net/doc/ai/nn/fully-connected/1988-lang.pdf#pa... works and doesn't diverge: if I'm understanding them right, they did warmup, so the gradients don't explode or vanish. So why doesn't that count? They have shortcut layers and a solution to exploding/vanishing gradients and it works to solve their problem. Is it literally 'well, you didn't use a gate neuron or fancy initialization to train your shortcuts stably, therefore it doesn't count'? Such an argument seems carefully tailored to exclude all prior work...

> Note again that a residual connection is not just an arbitrary shortcut connection or skip connection (e.g., 1988)[LA88][SEG1-3] from one layer to another! No, its weight must be 1.0, like in the 1997 LSTM, or in the 1999 initialized LSTM, or the initialized Highway Net, or the ResNet. If the weight had some other arbitrary real value far from 1.0, then the vanishing/exploding gradient problem[VAN1] would raise its ugly head, unless it was under control by an initially open gate that learns when to keep or temporarily remove the connection's residual property, like in the 1999 initialized LSTM, or the initialized Highway Net.

After reading Lang & Witbrock 1988 https://gwern.net/doc/ai/nn/fully-connected/1988-lang.pdf I'm not sure how convincing I find this explanation.

This apparently doesn't apply here, but in fact, pixels can be generated independently of each other. There are architectures where you can generate an arbitrary pixel or element of the image without generating the others; they are just implicit. See NeRFs or 'single-pixel GANs' or MAEs: eg https://arxiv.org/abs/2003.08934 https://arxiv.org/abs/2011.13775 https://arxiv.org/abs/2401.14391

Why is this possible? I tend to think of it as reflecting the ability to 'memorize' all possible data, and the independent generation is just when you 'remember' a specific part of a memory. The latent space is a Platonic object which doesn't change, so why should your generative process for materializing any specific point in the latent space have to? It's not surprising if you could generate arbitrary points from a function like 'y = mx + b' without generating every other point, right? It's just an atemporal mathematical object. Similarly with 'generating images from a random seed'. They too are just (complicated) functions mapping one number to another number.

(You might wonder if this is limited to images? It is not. In fact, you can generate even natural language like this to some degree: https://github.com/ethan-w-roland/AUNN based on my proposal for taking the 'independent generation' idea to a pathological extreme: https://gwern.net/aunn )

Requires a login?

I think OP is an instance that proves the point

this is a lazy, clumsy editing attempt, done through a document registration service which exists to prevent exactly this and yet, you have to be an experienced nerd (eg https://en.wikipedia.org/wiki/Matthew_Garrett has a doctorate and decades of software development experience) who will jump through a bunch of hoops to even begin to build a case beyond he-said-she-said. And he still doesn't have a settlement or criminal conviction in hand, so he's not even half done... Or look at the extensive forensics in the Craig Wright case just to establish simple things like that they were edited or backdated to a legally acceptable level.

Meanwhile, the original PDF edit in question took maybe 5 minutes with entry-level PDF tools.

One of the most surprising Gwern.net bug reports was from a compulsive highlighter who noted that the skip-ink implementation (which uses the old text-shadow trick, because frustratingly, the recently standardized skip-ink CSS still manages to fail at its only job and it looks awful) looked bad because of how browsers handle shadows and highlighting.

We had known about that (and it can't be fixed because browsers don't let you control the highlighting), but we had never imagined it'd be a problem because you'd only see it briefly when once in a while copy-pasting some text for a quote - right? I mean, why else would anyone be highlighting text? You'd only highlight small bits of text you had already read, so if it looked bad in places, that was fine, surely.

(Narrator: "It was not fine.")

Just another instance of Hyrum's law, I guess...

We decided to WONTFIX that because we can't easily fix that without making it uglier for users who don't abuse highlighting and are reading normally, which is almost everyone else.

Demonstrating that Rich Sutton was never really on the 'LLM bus' in the first place. Note the remarkable absence from the essay of language models & large language models from that essay despite BERT and GPT-2 and 'unreasonable effectiveness of data' etc. He only briefly mentions speech recognition. (Note also Sutton's general absence from LLM research, the Edmund Plan or switch from DeepMind to Keen Technologies as DeepMind was forced into LLM-centric research, and his published research since 2019's emphasis on small models and trying to fix their pathologies like catastrophic forgetting.)

> The bitter lesson is based on the historical observations that 1) AI researchers have often tried to build knowledge into their agents, 2) this always helps in the short term, and is personally satisfying to the researcher, but 3) in the long run it plateaus and even inhibits further progress, and 4) breakthrough progress eventually arrives by an opposing approach based on scaling computation by search and learning. The eventual success is tinged with bitterness, and often incompletely digested, because it is success over a favored, human-centric approach.

You could easily seem most LLM work as a dead end because it is about 'building knowledge into your agents' (eg. by paying data labelers billions of dollars total to supplement your scrapes), and not about 'search' (still a major open problem for LLMs - o1-style serial reasoning traces are obviously inadequate) or 'learning' (LLMs depend so heavily on the knowledge already encoded in so much data for them).

I don't have any particularly canonical reference I'd cite here, but self-preference bias in LLMs is well-established. (Just search Arxiv.)

They can, but they are known to have a self-favoring bias, and in this case, the error is so easily identified that it raises the question of why GPT-5 would both come up with it & preserve it when it can so easily identify it; while if that was part of OP's original inputs (whatever those were) it is much less surprising (because it is a common human error and mindlessly parroted in a lot of the 'scaling has hit a wall' human journalism).

It is lacking in URLs or references. (The systematic error in the self-reference blog post URLs is also suspicious: outdated system prompt? If nothing else, shows the human involved is sloppy when every link is broken.) The assertions are broadly cliche and truisms, and the solutions are trendy buzzwords from a year ago or more (consistent with knowledge cutoffs and emphasizing mainstream sources/opinions). The tricolon and unordered bolded triplet lists are ChatGPT. The em dashes (which you should not need to be told about at this point) and it's-not-x-but-y formulation are extremely blatant, if not 4o-level, and lacking emoji or hyperbolic language; hence, it's probably GPT-5. (Sub-GPT-5 ChatGPTs would also generally balk at talking about a 'GPT-5' because they think it doesn't exist yet.) I don't know if it was 100% GPT-5-written, but I do note that when I try the intro thesis paragraph on GPT-5-Pro, it dislikes it, and identifies several stupid assertions (eg. the claim that power law scaling has now hit 'diminishing returns', which is meaningless because all log or power laws always have diminishing returns), so probably not completely-GPT-5-written (or least, sub-Pro).

It is especially not obvious because this was written using ChatGPT-5. One appreciates the (deliberate?) irony, at least. (Or at least, surely if they had asymptoted, OP should've been able to write this upvoted HN article with an old GPT-4, say...)

That is unintentional and a bug in the dark-mode.

For dark-mode, we rely on https://invertornot.com/ to dynamically decide whether to fade or negate/invert. (Background: https://gwern.net/invertornot ) The service uses a small NN and is not always correct, as in these cases. Sorry.

I have filed them as errors with InvertOrNot, and will manually set them to invert.

The final section pounding the desk about how terrible ending the program was seems like it is oddly at variance with all the evidence OP had just laid out about how the program wasn't working well anymore and so wasn't actually financially a good idea. It's weird to quote a bunch of things like studies showing that 'internal R&D spending works worse than external for ROI' and then write a big moralizing sermonizing conclusion about how ending internal R&D is bad for profits and how terrible it is there's no 'patient capital' (capital which was plenty available before - what's the theory, investors stopped liking making money? insurance companies with century-long investment horizons ceased to exist? etc).

The mountain gave birth to a mouse.

Value of LLM writing advice: −$3,480; total value: −$3,500.

The initial tweet, 2020-03-05: https://x.com/joeld/status/1235652084264886272

> "I have been working on a reimagining of the blog idea for a few years, and it includes an idea (“series”) that is quite similar to blogchains. See this section of the design docs https://thelocalyarn.com/code/uv/scribbled/Basic_Notions.htm... and partial screen shot. It’s almost ready!"

My original response, in full:

> "One thought on the docs: if there's always a well-defined 'next', why not overload Space/PgDwn to proceed to the next node, GNU Info-like? At the very least, there should be a 'next' link at the bottom, not solely hidden away at the top.

> (Also, no one likes those silly 'st' ligatures.)

> As far as the current theyarn design goes: I like the use of typographic ornaments as a theme, but the colors are confusing. (What does orange vs red vs green denote?) And the pilcrow? Sometimes it's at the beginning of articles (redundant with the hr), sometimes not?

> Hrs seem overused in general, like the (busy) footer. Appending notes chronologically is interesting but confusing, both date/where they begin/end. Are the caps deliberately not small caps? Full-width images would be useful for photos. Be interested to see the new one finished."

I consider these criticisms reasonable, accurate, and constructive, milquetoast even, and stand by them. I see no difference from the many other site critiques I have made over the years (eg https://www.lesswrong.com/posts/Nq2BtFidsnhfLuNAx/announcing... ), which are usually received positively, and I think this is a 'you' problem, especially after reading your other comments. And I will point out that you made no reply to my many concrete points until you decided to write this HN comment 5 years later.

(I have taken the liberty of adding a link to your top-level comment to the end of the existing thread, for context/updating.)

> But responding as though this is an attack to be countered and defeated further illustrates why I had doubts about your suitability as a creative advisor.

This is a remarkable way to characterize this conversation.

Might be thinking of https://www.dwarkesh.com/p/gwern-branwen#%C2%A7influencing-t...

> Edit: if someone can explain why this was instantly downvoted I would genuinely appreciate knowing where I went wrong here

Alluding to some conversation which supposedly demonstrates my unsuitability for such a role, while pointedly refusing to link it or describe it any detail which could be judged (to the point where I, the person in question, have no idea what misdeeds you are talking about), is not credible and reeks of, one might say, 'very unconstructive crap-on criticism', and people might be understandably reluctant to upvote it.

If you think my criticisms were that bad, then link the tweets in question - and, since my account is currently locked, I will happily copy them all out here so everyone can read them and judge for themselves. I am not afraid.

> But it felt like a lot of the wording is speculative or hyperbolic in nature and aggressively tries to paint Volvo in a bad light. For example:

Part of it is that he clearly used ChatGPT or Claude to write the prose. (I really should not have to explain how, despite not reading the OP at all, your example quote alone establishes that. You see this kind of hyperbolic unordered list/checklist all the time now. This seems like more of a Claude tic, but could also be ChatGPT due to sheer base rates.)

Being sycophantic and ordered to write polemic, a LLM'll go overboard.

Because it has many of the typical 4o stylistic tics like 'it's not X, it's Y' or enumeration or the em dashes, or the twist ending.

It's not 100% unedited ChatGPT and far from the most blatant instance that has caught my eye (they've started showing up in the New York Times and New Yorker as well, have you noticed that?), but certainly sounds like that was used: "Writing compels us to think — not in the chaotic, non-linear way our minds typically wander, but in a structured, intentional manner." "This is not merely a philosophical observation; it is backed by scientific evidence." "Importantly, if writing is thinking, are we not then reading the ‘thoughts’ of the LLM rather than those of the researchers behind the paper?" "overcoming writer’s block, provide alternative explanations for findings or identify connections between seemingly unrelated subjects."

(Note that this is particularly ironic because as the op-ed notes, if they did use it, they are required by Nature to disclose this... https://www.nature.com/articles/d41586-023-00191-1 But of course, how would anyone ever prove they did so? You know how difficult it is to get Nature to retract even blatant fraud.)

New HN metric: how many comments do you have to scroll down on a LLM-edited/written submission before someone points it out? Triple bonus if the submission is about LLM writing and the need to think for yourself without a LLM, and still no one has pointed this out. Current page score: 333 points.

No. The tanks problem is when you are picking up a genuine signal in the data. If you collected more data exactly the same way, then it would continue to predict well on this new data; and many other model architectures would also pick up the signal (because it's genuine).

This is the obvious explanation for their initial results, and one of the first things I said when I saw the preliminary results: "how do you know there isn't some subtle association between 'eagle' and $arbitrary_small_integer that you are just ignorant of but the superhumanly knowledgeable LLMs have learned about?"

But then the later experiments rule that out, in part by showing that it doesn't transfer across different models (ie. initializations).

It's an open question how far this will transfer. Given the local basin/optima approach, and the incestuous nature of AI outputs + training, it's entirely possible that you could start to see 'lineages' of AIs (often undeclared, eg based on abusing APIs for distillation, and maybe unknown even to the creating entity if people/AI inside it are lying or hustling) where there is a lot of acausal coordination going on due to this.

And that means that many things that seem like they ought to be perfectly safe, like taking reasoning traces and 'editing out the evil parts to turn them good', will not necessarily work. (Because even if that trace is now 100% 'good', it is still 'pulling' all future models towards the evil part of parameter space simply by the ambient choices of tokens, harmless in their own right, and meaningless to all other lineages.)

> there's no loophole where they'd specifically freeze it out of participating in tender offers.

Again, who said anything about a 'specific loophole'? Needing permission to participate in a tender (which is the only way to sell) is not a 'loophole', and the threat is always there on the table. So again: how was that 'fixed'? Should I interpret your comment as implying that the tender threat of being frozen out is not fixed?

Certainly your fellow OA employee in the other comment doesn't seem to think that it's not on the table, because he is arguing that the threat is fine and harmless because it's never been exercised, which would seem to imply that it's still there...

>135 comments: https://news.ycombinator.com/item?id=44578070

Then how was that "fixed"?

I think the fact that you consider that a defense is a good illustration of why I had to ask that question. ("Yes, the gun is on the table, but the trigger has never been pulled. Sorry to ruin your witch hunt.")