Hacker News Reader

In my experience, "the bitter lesson" is just wrong. Yes, a hypothetical AI model that has a suitable architecture so that it can memorize your handcrafted features can improve upon them, if it is perfectly trained. But that's a huge if. There's a reason that large AI models only work with some architectures and you need to randomly initialize tens to hundreds of times and you need exactly the right optimizer with the right hyperparameters and then the right training data in the right order... and the reason is that generic function approximators get stuck in a local minima incredibly easy.

People used to write handcrafted features to limit the optimization base area. Now people use specific pretraining data and initialization functions to do the same. It's just a different way to express the same constraints.

In short, the "bitter lesson" assumes a change that did not happen yet.

>and the reason is that generic function approximators get stuck in a local minima incredibly easy.

The key insight in massively over parameterized models (aka LLMs) has been that all the local minima are very similar. Picking one over the other doesn't actually benefit you that much.

Absolutely not! LLM training quite easily gets stuck. People call it "divergence" or "loss spike". See for example:

https://dl.acm.org/doi/10.5555/3524938.3525354

"Gradient optimization with attention layers can be notoriously difficult requiring tricks such as learning rate warmup to prevent divergence.“

"we then propose a new weight initialization scheme with theoretical justification, that enables training without warmup"

Or "Initialization of Large Language Models via Reparameterization to Mitigate Loss Spikes"

https://arxiv.org/html/2410.05052v1

If all local minima were similar, then these papers would represent a lot of wasted effort. Or said in a different way: If labs like DeepMind are willing to pay for large teams that are trying to optimize initialization, then surely this represents a real existing problem.

There are basically no people who have done pre-training of LLMs.

What you're describing is absolutely the case, but everyone not in a top 100 AI lab would have never had the displeasure of having to deal with that.

That said I strongly suggest everyone follow along with a GPT2 tutorial like: https://github.com/karpathy/minGPT to get a feel for what the big boys have to deal with. Any $100,000 training run can end in failure for no reason what so ever because you just got unlucky with your random seed.

If you want something that is both SOTA and cheap: https://huggingface.co/blog/modernbert you can get from pre-training to fine tuning for a couple hundred dollars with a model that is better than anything available to the public. Bert models are so under researched that you legitimately push the state of the art for them on a work station.

Also feel free to drop a line at my profile's address.

nit: a divergent training run is somewhat separate from the question of local minima. The latter observes the loss curve stabilize and asks 'is the _best_ (loss-minimizing) set of parameters for this loss/data/model/hypers/seed/etc?'. The former observes the loss curve explode, and asks for another 10k GPU hours to try again with a different choice of loss/data/model/... .

They're definitely related in some instances e.g. 'flat' vs 'steep' local minima with comparable losses.

> There's a reason that large AI models only work with some architectures and you need to randomly initialize tens to hundreds of times and you need exactly the right optimizer with the right hyperparameters and then the right training data in the right order..

GPT4 used the tensor programs (Greg Yang,now at X. Ai) research to effectively transfer hyper parameters from smaller models where experiments can go fast to the larger one in a predictable way and got a very smooth loss curve throughout the training.

But that's OPs point. You need to care about those hyper parameters. Good on them for finding them in smaller models, but they still had to find them, because without those hyper parameters pre-training is at best unstable and at worst divergent.

Also small here means hundreds of millions to billions of parameters - that's still larger than any model pre 2020.

Prior to that work on hyperparameter transfer though things would go wrong much more often training the big expensive model and require restarts back to checkpoints and retuning parameters much more often though. Yeah it's still a lot of fiddling around but it is orders of magnitude less expensive.

But that work also went far beyond transfer of fiddling around and mathematically solved a lot of stuff with avoiding overfitting, great video overview of it here:

https://www.youtube.com/watch?v=1aXOXHA7Jcw

In my opinion, you've grossly mischaracterized and misunderstood the bitter lesson. The bitter lesson is not saying that brute force will win out against better algorithms, it's saying that simple, efficient algorithms, in a theoretical sense, will win out against intricate and specialized algorithms.

That is, it's saying that worrying about the constant factor in the "big-Oh" is doomed to failure. My reading is that it's not saying that we should abandon polynomial time algorithms in favor of exponential ones, it's saying we should focus on polynomial time algorithms that are conceptually simple and let Moore's law take care of the constant factor differences between intricate, complex and "clever" algorithms. It's saying that Moore's law will obliterate the constant factor differences in algorithms, and potentially the small polynomial exponent differences, not that we shouldn't care about exponential time complexity.

The "Bitter Lesson" blog post came out during a time when people were debating whether simple algorithms on large datasets could win out against more complex algorithms with intricate domain knowledge. That is, did AGI, vision, language, translation, speech, etc. need intricate and deep domain knowledge or was it good enough to get coarser and "stupider" algorithms, that were still theoretically efficient/polynomial time efficient algorithms.

I can see why you would think this way and I certainly don't claim to have deep insight into Sutton to claim whether my reading is more correct than yours. From my perspective, the good faith reading of Sutton's article is talking about constant factor differences in algorithms, not in differences between polynomial time and exponential time worst case complexities.

I wonder if an analogy with graphics would be that "simple algorithms that leveraged compute" would not be "brute force everything with 1000 gpus + ray tracing", it might be a pixel shader that simulated water taking the state of the art a bit further.

The bitter lesson is about the balance of human ingenuity and compute being thrown at the problem. We've seen a few years of LLM compute being scaled up 10x every year, but this is hitting limits (fabs), and we will see more human effort, as it becomes compartively cheaper.

Also the current crop of models are inherently limited. Even for something as simple as following a JSON schema, models alone are not good enough [0]

Of course as the Moore law refuses to die, we'll continue seeing 1.5-2x or so every year, but that's far from 10x.

[0] https://openai.com/index/introducing-structured-outputs-in-t... - see plot

>Of course as the Moore law refuses to die, we'll continue seeing 1.5-2x or so every year, but that's far from 10x.

This is another one of those anec-data throw away sentences that take thousands of words to disprove - with a lot of graphs - that no one reads.

More hot takes: Moores law has been effectively dead since the Pentium 4 on CPUs. It's been dead on GPUs since 2020. Right now we're not seeing a 1.5-2x grow of compute per year. We've seen zero growth for 5. The only way that GPUs have gotten faster is by running ever hotter, and building out a trillion dollars worth of data centers.

No one cares because the current hotness in AI is transformers which are memory bound in both training and inference. If someone manages to get diffusion models to become the next hotness all of a sudden everyone will realize this is a problem since those are compute bound by a huge margin and current gen GPUs are fire hazards when ran at 100% utilization for weeks on end.

Yeah, my experience is that the rapid growth of computing capabilities of my childhood more or less ended in 2013: the desktop I built then is fine for most of my uses and the significant performance improvements since then haven’t been CPUs getting noticeably faster but increases in the speed of disks (SSDs) and growth of memory sizes and network speeds.

the only escape from the Black Pill of AI ruining society is we hit some hard limits and technology stops progressing at such a rapid rate, tapers, and gives humanity a bit of time to adapt culturally.

wat... my 9950x cpu that i just bought is way faster than the similarly priced cpu i bought 6 years ago.

The difference is night and day. What are you talking about.

the equivalent chip 6 years ago was the 3950x which is the same number of cores clocked ~20% slower. if you add on ~30% ipc (for avx-512 and some general cleanups), you get a 60% speedup in 6 years which is way below the ~10x speedup between the similarly spaced Pentium 4 and Sandy bridge

how about power consumption, cache size, etc?

what i know is that in my particle simulations and multithreaded compilation tasks it is easily 2-10x faster. where before i could choke on 1 million particles, now im choking on 30mill.

i specifically bought this processor for its rust compilation times.

>what i know is that in my particle simulations and multithreaded compilation tasks it is easily 2-10x faster.

If Moore's law still worked like it did between 1975 and 2005 you'd be getting a 16x performance boost in _single_ threaded applications.

The fastest single threaded, non overclocked, commercially available cpu was the Intel i3 7350k.

I haven't seen anything bench faster than a system set up for that machine on a business board.

I think now with advancements such as AVX (the newer versions that chips like atoms don't have) the 7350k can be viewed as a power hungry pig.

I'm not sure though. The systems I purpose built using those CPUs still work great, no real significant performance issues even with current software and web. And they can use a single battery UPS.

I still have a 16 core 5950x, that is 10% faster than the 40 core xeon box it replaced, at 1/4 the wattage at the wall as my main machine.

Its closer to 2X faster on Chromium compile benchmark, at least the 9950X3D, and even bigger benefits for X3D in other use cases than code compiling, 9950X3D is only a few percent better than 9950X for that.

>More hot takes: Moores law has been effectively dead since the Pentium 4 on CPUs.

AMD is offering double digit percentage increases in IPC every generation. What you were saying might have been valid in early 2017, back when Intel was cranking up the clock speeds for a 5% increase, but when Zen came out suddenly AMD started delivering again.

It's much higher than 1.5x-2x, both scale up and scale out. Moving onto fp4 alone will offer a huge speedup.

Agree (though that's perhaps not a charitable interpretation of TBL). Prematurely articulated principles can do a lot of damage.

Btw I found the blog post to be one of the lowest quality one in terms of information content posted on HN - almost like it was written by chatgpt or something.

I don't feel like the bitter lesson becomes applicable until you've demonstrated some initial degree of success with your technique on a single machine/GPU/CPU/thread. If you cannot make it work in practical environments, it's going to be an uphill battle the entire way.

This is why I've moved toward CPU-only techniques in my experimentation. Being able to execute arbitrary UTMs with high performance provides a significantly richer computational landscape to work with than matrix multiplication. I am perfectly happy with something taking longer as if it's on a linear basis. I.e., adding another CPU provides ~2x search speed. I am NOT happy with ideas like taking a 100x hit on token generation rate because half my parameters are paged out to disk at any given moment due to not having a step-size amount of vram.

The rigidity of the GPU solution stack makes exploration of clever techniques largely a slap on the wrist experience. Anything with rapid control flow changes is verboten for a fancy pants cluster. The latency domain of L1 cache in the CPU is impossible to compete with if you need to serialize all of your events. I strongly believe that control flow is where the magic happens. This is where you can cut through 100 million parameters of linear math bullshit and solve the problem with a lookup table and 2 interpreter cycles. You get about half a billion of these cycles to work with per second per thread, so there is a lot of room to play with ideas.

Meanwhile in robotics everyone is glad that computers are getting faster.

There are a lot of things that used to be impossible to do inside a 1000 Hz control loop.

>What do we do with the problems that are too hard to solve with the limited compute that we have? Lie down for 80 years and wait for compute to catch up? Or solve a smaller problem using specialized tricks that don't require a $10B super computer to build?

Solving a smaller problem using specialized tricks has gone nowhere in robotics. Almost all the advancements in robotics control happened in the last ten years as a result of computers getting faster.

We are very close to the cusp of non linear MPC becoming a solved problem for up to 64 degrees of freedom and a horizon of 20 time steps at 1000Hz, but we aren't there yet. It would definitely be possible with an ASIC built for MPC.

>The bitter lesson is nothing of the sort, there is plenty of space for thinking hard, and there always will be.

The bitter lesson doesn't say that human ingenuity is worthless. It guides it in a useful direction. A lot of human ingenuity was put into compute scaling solutions for transformers.

"The bitter lesson" didn't merely claim that a sufficiently large amount of compute would obsolete an engineered solution. Its claim was far stronger: the time it takes for the compute growth to catch up with the hand-engineered solution is so short that the investment in the latter won't pay off in the sense of a researcher's personal career investment, or in the sense of a big-tech R&D effort ROI.

You may well dispute such a claim, of course. Would be interesting to read your thoughts if you are willing to share them.

> The raw compute needed to brute force any problem can only be known after the problem is solved. There is no sane upper limit to how much computation, memory and data any given task will take and humans are terrible at estimating how hard tasks actually are. We are after all only 60 years late for the undergraduate summer project that would solve computer vision.

I feel like you’re conflating conceptual difficulty and computational difficulty

I appreciate this comment. I'm currently working hard on something seemly straightforward (address matching) and sometimes I feel demotivated because it feels like whatever progress I make, the bitter lesson will get me in the end. Reading your comment made me feel that maybe it's worth the effort after all. I have also taken some comfort in the fact that current LLMs cannot perform this task very well.

Can you provide more about the problems in address matching and what you are trying to solve?

Do you mean street address matching? Isn’t that already solved? (excuse the naive question)

I don’t know of any open source solution. (I work in mapping)

The problem is that people write addresses down in different ways (I'm in the UK)

You will typically have a master/canonical list of addresses from an official source. In the UK that's Ordnance Survey's AddressBase.

You will then have 'messy addresses' that humans have written down. For simple addresses, they'll often be the same as the master version (e.g. 5 Rainbow Road, Hemel Hempstead, AB1 2BC).

But there are many harder addresses that exhibit lots of variations, especially flats and subunits.

These may all be the same: Flat A 1 High Street vs 1A High Street vs Basement Flat, 1 High Street

There no guarantee there will be a number in the address. In: 'THE OLD FARM COTTAGE PAD FARM BADGERCROFT ROAD PIKING', 'THE' and PAD FARM' may be missing, which doesn't seem like a problem until you find out there's also a PAD FARM COTTAGE on Badgercroft Road. There's no guarantee tokens will be in the same order.

FWIW, my work is open source, and it's here: https://github.com/RobinL/uk_address_matcher

Thanks for explaining. Sounds messy indeed. Wasn’t there this startup trying to offer an alternative? Three words to describe a location or something. But of course it’s not really easy to change an address system, so using AI to make the existing system less messy might be more pragmatic.