Comment by QuesnayJr | Hacker News Reader

Comment by QuesnayJr | Hacker News Reader

Parent Full thread

I'm starting to think that half the commenters here don't actually know what "The Bitter Lesson" is. It's purely a statement about the history of AI research, in a very short essay by Rich Sutton: http://www.incompleteideas.net/IncIdeas/BitterLesson.html It's not some general statement about software engineering for all domains, but a very specific statement about AI applications. It's an observation that the previous generation's careful algorithmic work to solve an AI problem ends up being obsoleted by this generation's brute force approach using more computing power. It's something that's happened over and over again in AI, and has happened several times even since 2019 when Sutton wrote the essay.

That essay is actually linked in the lead:

> As it's been pointed out countless times - if the trend of ML research could be summarised, it'd be the adherence to The Bitter Lesson - opt for general-purpose methods that leverage large amounts of compute and data over crafted methods by domain experts

But we're only 1 sentence in, and this is already a failure of science communication at several levels.

1. The sentence structure and grammar is simply horrible

2. This is condescending: "pointed out countless times" - has it?

3. The reference to Sutton's essay is oblique, easy to miss

4. Outside of AI circles, "Bitter Lesson" is not very well known. If you didn't already know about it, this doesn't help.

I think most people have read it and agree it makes an astute observation about surviving methods, but my point is that now we use it to complain about new methods that should just skip all that in between stuff so that The Bitter Lesson doesn't come for them. At best you can use it as an inspiration. Anyway, this was mostly a complaint about the use of "The Bitter Lesson" in the context of this article, it still deserves credit for all the great information about tokenization methods and how one evolutionary branch of them is the Byte Latent Transformer.

Reply on news.ycombinator.com