joefourier

With modern training techniques, RNNs (not just linear SSMs, potentially even vanilla LSTMs) can scale just as well as transformers or even better when it comes to enormous context lengths. Dot-product attention has better performance in a number of domains however (especially for exact retrieval) so the best architectures are likely to remain hybrid for now.

> 2017’s Attention is All You Need was groundbreaking and paved the way for ChatGPT et al. Since then ML researchers have been trying to come up with new architectures, and companies have thrown gazillions of dollars at smart people to play around and see if they can make a better kind of model. However, these more sophisticated architectures don’t seem to perform as well as Throwing More Parameters At The Problem. Perhaps this is a variant of the Bitter Lesson.

This is not true and unfortunately this significantly reduced the credibility of this article for me. Raw parameter counts stopped increasing almost 5 years ago, and modern models rely on sophisticated architectures like mixture-of-experts, multi-head latent attention, hybrid Mamba/Gated linear attention layers, sparse attention for long context lengths, etc. Training is also vastly more sophisticated.

The Bitter Lesson is misunderstood. It doesn't say "algorithms are pointless, just throw more compute at the problem", it says that general algorithms that scale with more compute are better than algorithms that try to directly encode human understanding. It says nothing about spending time optimising algorithms to scale better for the same compute, and attention algorithms and LLMs in general have significantly advanced beyond "moar parameters" since the time of Attention is All You Need/GPT2/GPT3.

Ever hit your daily limit on Claude Code and saw how expensive it is to pay per token?

The dotcom bubble burst and 26 years later we’re all hopelessly addicted to the internet and the top companies on the stock market are almost all what would have been called “dotcoms” then.

The railroad bubble burst in 1846 not because trains were a dead end - passenger number would increase more than 10x in the UK in the following 50 years.

It’s absolutely not winner take all. LLMs have become a commodity and the cost of switching models is essentially nil.

Even if ChatGPT has brand recognition amongst lay people, your grandparents aren’t the ones shelling out $200/mo for a Claude code subscription and paying for extra Opus tokens on top of that. Anthropic’s revenue is now neck and neck with OpenAI, but if tomorrow they increased the price of Opus by 5x without increasing its capabilities, many would switch to Gemini, GPT 5.4, Cursor, or any cheap Chinese model. In fact I know many engineers that have multiple subscriptions active and switch when they hit the rate limits of one, precisely the tools are so interchangeable.

At some point it could even become cheaper to just buy 8x H100s and host Qwen/Deepseek/Kimi/etc yourself if you’re one of those companies paying $3k/mo per engineers in tokens.

Then what do you call RAG done well? You need a term for it.

> And when you hear someone saying "we use RAG here" 95% of the time this is exactly what they mean.

That's just Sturgeon's law in action. 95% of every implementation is crap. Back in the 90s, you might have heard "we use OOP here" and come to a similar conclusion, but that doesn't mean you need to invent a new word for doing OOP properly.

> But agentic RAG is fundamentally different.

From an implementation POV, absolutely not.

I've personally gradually converted a dumb semantic search to a more fully featured agentic RAG in small steps like these:

- Have a separate LLM call write the query instead of just using the user's message. - Make the RAG search a synthetic injected tool call, instead of appending it to the system prompt. - Improve the search endpoint by using an LLM to pre-process the data into structured chunks with hierarchical categories, tags, and possible search queries, embedding the search queries separately from the desired information (versus originally just having a raw blob). - Have the LLM be able to search both with a semantic sentence, and a list of tags. - Have the LLM view and navigate the hierarchy in a tree-like manner. - Make the original LLM able to call the search on its own instead of being automatically injected using a separate query rewriting call, letting it search in multiple rounds and refine its own queries.

When did the system go from RAG to "not RAG"? Because fundamentally, all you need to do to make an agentic RAG is to have the LLM be able to write/rewrite its own search queries (possibly in multiple passes) as opposed to just passing the user's messages(s) directly.

> What if your inquiry needs a combination of multiple sources to make sense? There is no 1:1 matching of information, never.

I don't see the problem if you give the LLM the ability to generate multiple search queries at once. Even simple vector search can give you multiple results at once.

> "How many cars from 1980 to 1985 and 1990 to 1997 had between 100 and 180PS without Diesel in the color blue that were approved for USA and Germany from Mercedes but only the E unit?"

I'm a human and I have a hard time parsing that query. Are you asking only for Mercedes E-Class? The number of cars, as in how many were sold?

I agree with you that simple vector search + context stuffing is dead as a method, but I think it's ridiculous to reserve the term "RAG" for just the earliest most basic implementation. The definition of Retrieval Augmented Generation is any method that tries to give the LLM relevant data dynamically as opposed to relying purely on it memorising training data, or giving it everything it could possibly need and relying on long context windows.

The RAG system you mentioned is just RAG done badly, but doing it properly doesn't require a fundamentally different technique.

Some previous techniques for RAG, like directly using a user message’s embedding to do a vector search and stuffing the results in the prompt, are probably obsolete. Newer models work much better if you use tool calls and let them write their own search queries (on an internal database, and perhaps with multiple rounds), and some people consider that “agentic AI” as opposed to RAG. It’s still augmenting generation with retrieved information, just in a more sophisticated way.

Mamba doesn't assume auto-regressive decoding, and you can use absolutely use it for diffusion, or pretty much any other common objective. Same with a conventional transformer. For a discrete diffusion language model, the output head is essentially the same as an autoregressive one. But yes, the training/objective/inference setup is different.

But why? That's like trying to determine which car is faster by looking at only at the rpm.

You don't even need to go into the pipeline details. The 9800X3D has 8x more L2 cache, 6x more L3 cache, 2x the memory bandwidth than the now 8 years old i9 9900K. 3D V-cache is pretty cool.

I'm not from that generation so that's a bit hard for me to understand. Even if you used a closed-source C compiler, wouldn't you still have been able to look at the header file, which would have been pretty self-explanatory?

E.g.

void qsort(void* base, size_t nmemb, size_t size, int (compar)(const void , const void* ));

And surely if you bought a C compiler, you would have gotten a manual or two with it? Documentation from the pre-Internet age tended to be much better than today.

Oh I'm happy this has a name now! Even if it's quite silly.

> HN is an interesting place.

How is it interesting that in a forum with thousands of active users, someone posted a comment that disagreed with other comments from 6 years ago?

Even in this thread, there's many different opinions being expressed.

Surely it's more of a spectrum? From a CPU, to a TPU, to a chip that hardwires softmax attention but lets you store arbitrary weights, to one that hardwires the weights directly.

That depends what kind of ASIC you’re talking about. Cerebras can run models like GLM 4.7 with 355B parameters.

The first professional commercial 4K camera came out over 23 years ago, and the first smartphones and camcorders capable of 4K video were back in 2013.

The Macbook Neo has a 2.5x higher multi-core Geekbench score compared to the i7-4960X's, the top consumer CPU of 2013 (which could handle 4K video editing in h264), and its single-core performance is 5x higher. Plus, I'm 99% sure the MacBook Neo has a dedicated video decoding ASIC anyway.

Fine-tuning still makes sense for cost/latency-sensitive applications. Massive context windows drastically slow down generation, and modern models' performance and instruction following ability relies heavily on a reasoning step that can consume orders of magnitude more tokens than the actual response (depending on the application), while a fine-tuned model can skip/significantly reduce that step.

Using the large model to generate synthetic data offline with the techniques you mentioned, then fine-tuning the small model on it, is an underrated technique.

You don't need 8GB of RAM or less to have memory issues. Cursor + Claude Code + Slack + Discord + Spotify + a few Docker containers + YouTube and a few browser tabs is enough to overwhelm a MacBook with 24GB of RAM.

Right now on my machine, 5 whole docker containers, including two DBs and 3 dev servers, are taking up less RAM than Cursor, a glorified text editor.

And have you looked at RAM prices lately? It's possible that 8GB is all some people can afford.

What's your definition of intelligence? If you exclude LLMs, you might have to exclude quite a few humans as well.

When did AGI start meaning ASI?

LLMs are artificial general intelligence, as per the Wikipedia definition:

> generalise knowledge, transfer skills between domains, and solve novel problems without task‑specific reprogramming

Even GPT-3 could meet that bar.

Tensorflow is largely dead, it’s been years since I’ve seen a new repo use it. Go with Jax if you want a PyTorch alternative that can have better performance for certain scenarios.

You can actually generate surprisingly coherent text with minimal finetuning of BERT, by reinterpreting it as a diffusion model: https://nathan.rs/posts/roberta-diffusion/

I don’t see a useful definition of LLM that doesn’t include BERT, especially given its historical importance. 340M parameters is only “small” in the sense that a baby whale is small.

Well in that case wouldn't that be millions of ASIs, each with contradictory goals?

I'm not saying that ASI isn't an existential threat, just that it probably won't present itself like the fanciful sci-fi scenario of a singular intelligence suddenly crossing a magic threshold and being able to take over the world. Most likely it will be some scenario we won't have predicted, the same way hardly anybody predicted LLMs.

> No, it would have been called AI. A decade ago most people were not familiar with AGI as a term, that just got popularised because AI was taken over to be basically what we used to call ML.

Define "most people", I don't think the average user of ChatGPT is familiar with the term AGI even now, but it's been used in the AI/ML community for multiple decades. I remember reading about the distinction between general and narrow AI around 2010 as an enthusiast. "Strong" vs "weak" AI were also used although with essentially the same definition, although they're less common terms nowadays.

We are already at AGI. I don’t know how you can argue that LLMs don’t meet the definition of general artificial intelligence, as opposed to narrow AI like chess engines, image classifiers, AlphaGo or self driving cars, which are trained with one objective and cannot even possibly be applied to any other task.

People have just moved the goalposts, imagine explaining Opus 4.6’s capabilities to someone even 10 years ago, it would definitely have been called AGI.

I highly doubt there will be a point where everyone will agree that we’ve achieved ASI, there will always be a Gary Marcus type finding some edge case where it performs poorly.

ASI still runs at finite speed and is limited by its hardware, and speed of its interactions with the real world. It won’t be able to recursively improve itself overnight if it only generates 10 tokens per seconds, and a second company could very well train one of its own before the first one has time to do much.

Oh yes you're correct, imaging would be the correct term for what's happening I think (aliasing is high -> low and imaging is low -> high)?

The author is correct in that agents are becoming more and more capable and that you don't need the IDE to the same extent, but I don't see that as good. I find that IDE-based agentic programming actually encourages you to read and understand your codebase as opposed to CLI-based workflows. It's so much easier to flip through files, review the changes it made, or highlight a specific function and give it to the agent, as opposed to through the CLI where you usually just give it an entire file by typing the name, and often you just pray that it manages to find the context by itself. My prompts in Cursor are generally a lot more specific and I get more surgical results than with Claude Code in the terminal purely because of the convenience of the UX.

But secondly, there's an entire field of LLM-assisted coding that's being almost entirely neglected and that's code autocomplete models. Fundamentally they're the same technology as agents and should be doing the same thing: indexing your code in the background, filtering the context, etc, but there's much less attention and it does feel like the models are stagnating.

I find that very unfortunate. Compare the two workflows:

With a normal coding agent, you write your prompt, then you have to at least a full minute for the result (generally more, depending on the task), breaking your flow and forcing you to task-switch. Then it gives you a giant mass of code and of course 99% of the time you just approve and test it because it's a slog to read through what it did. If it doesn't work as intended, you get angry at the model, retry your prompt, spending a larger amount of tokens the longer your chat history.

But with LLM-powered auto-complete, when you want, say, a function to do X, you write your comment describing it first, just like you should if you were writing it yourself. You instantly see a small section of code and if it's not what you want, you can alter your comment. Even if it's not 100% correct, multi-line autocomplete is great because you approve it line by line and can stop when it gets to the incorrect parts, and you're not forced to task switch and you don't lose your concentration, that great sense of "flow".

Fundamentally it's not that different from agentic coding - except instead of prompting in a chatbox, you write comments in the files directly. But I much prefer the quick feedback loop, the ability to ignore outputs you don't want, and the fact that I don't feel like I'm losing track of what my code is doing.