Hacker News Reader

gdiamoson The case for the return of fine-tuning

We used the best models available and went from the Pythia/gpt2 to Deepseek generations.

One annoying part was switching to new and better models that came out literally every week.

I don’t think it substantially changes anything. If anything I think the release of more advanced models like qwen-next makes things like fp4, moe, and reasoning tokens an even higher barrier of entry.

gdiamoson The case for the return of fine-tuning

How can you hire enough people to scale that while making the economics work?

Why would they join you rather than founding their own company?

gdiamoson The case for the return of fine-tuning

Try doing a head to head comparison using all LLM tricks available including prompt engineering, rag, reasoning, inference time compute, multiple agents, tools, etc

Then try the same thing using fine tuning. See which one wins. In ML class we have labeled datasets with breeds of dogs hand labeled by experts like Andrej, in real life users don’t have specific, clearly defined, and high quality labeled data like that.

I’d be interested to be proven wrong

I think it is easy for strong ML teams to fall into this trap because they themselves can get fine tuning to work well. Trying to scale it to a broader market is where it fell apart for us.

This is not to say that no one can do it. There were users who produced good models. The problem we had was where to consistently find these users who were willing to pay for infrastructure.

I’m glad we tried it, but I personally think it is beating a dead horse/llama to try it today

gdiamoson The case for the return of fine-tuning

They will hire anyone who can produce a model better than GPT5, which is the bar for fine tuning

Otherwise, you should just use gpt5

Preparing a few thousands training examples and pressing fine tune can improve the base LLM in a few situations, but it also can make the LLM worse at other tasks in hard to understand ways that only show up in production because you didn’t build evals that are good enough to catch them. It also has all of the failure modes of deep learning. There is a reason why deep learning training never took off like LLMs did despite many attempts at building startups around it.

Andrej karpathy has a rant about it that captures some of the failure modes of fine tuning - https://karpathy.github.io/2019/04/25/recipe/

gdiamoson The case for the return of fine-tuning

Just coming out of founding one of the first LLM fine tuning startups - Lamini - I disagree

Our thesis was that fine tuning would be easier than deep learning for users to adopt because it was starting from a very capable base LLM rather than starting from scratch

However, our main finding with over 20 deployments was that LLM fine tuning is no easier to use than deep learning

The current market situation is that ML engineers who are good enough at deep learning to master fine tuning can found their own AI startup or join Anthropic/OpenAI. They are underpaid building LLM solutions. Expert teams building Claude, GPT, and Qwen will out compete most users who try fine tuning on their own.

RAG, prompt engineering, inference time compute, agents, memory, and SLMs are much easier to use and go very far for most new solutions

gdiamoson Recursive Language Models (RLMs)

Recursion is so popular in computing that this term “recursive language model” is heavily overloaded

It was even before the rise of LLMs

The authors may want to consider a more specific name

gdiamoson How to inject knowledge efficiently? Knowledge infusion scaling law for LLMs

I wonder if this depends on what is inside the domain specific data.

I’m happy to see ML papers on hacker news.

gdiamoson DARPA project for automated translation from C to Rust (2024)

They should make the challenge to translate all of Linux and then maintain it

gdiamoson LLM Observability in the Wild – Why OpenTelemetry Should Be the Standard

LLM app telemetry is important, but I don’t think we have seen the right metrics yet. Nothing has convinced me that they are more useful than modern app telemetry

I don’t think tool calls or prompts or rag hits are it

That’s like saying that C++ app observability is about looking at every sys call and their arguments

Sure, if you are the OS it’s easy to instrument that, but IMO I’d rather just attach to my app and look at the logs

gdiamoson Reports: EA set to be sold to private investors for up to $50B

Some EA IP I remember

Battlefield

Mass effect

Dragon age

Madden

The sims

Titanfall

gdiamoson OpenAI and Nvidia announce partnership to deploy 10GW of Nvidia systems

It forces us to confront a question.

How much investment and prioritization in scaling laws is justified?

gdiamoson Evals in 2025: going beyond simple benchmarks to build models people can use

How can the community tell if models overfit to these benchmarks?

gdiamoson LLM-Deflate: Extracting LLMs into Datasets

LLMs have finite entropy (it is related to their training loss) and training typically doesn’t store the residuals.

Some compression methods use LLMs internally and also store the residuals, making them lossless.

gdiamoson LLM-Deflate: Extracting LLMs into Datasets

There are some fun early theoretical ML papers on this topic.

They prove that it is possible to fully clone a brain based on this method.

I think one could theoretically estimate how many queries you would need to make to do it. The worst case is proportional to the number of parameters of the model, i.e. at least 10^15 for a human. At one minute per spoken sample, that comes out to about 2 billion years to clone one human.

I suspect it is not practical without advancements in neural link to increase the bandwidth by billions of times.

I personally like playing around with empirical methods like this blog post to understand the practical efficiency of our learning algorithms like back prop on transformers.

I also try not to invest too much effort into this topic given the ethical issues.

gdiamoson LLM-Deflate: Extracting LLMs into Datasets

It would be an expensive experiment to perform.

I wonder how you could do it more efficiently?

gdiamoson LLM-Deflate: Extracting LLMs into Datasets

I think it would be interesting to deflate out to a huge dataset and see where this happens.

Certainly it will occur as the generated data exceeds the original, eg after 1-10T tokens.

I think you could also do this faster by moving down the tree in a depth first manner.

Typically I use this for knowledge transfer, style transfer, catastrophic forgetting mitigation, etc and so I don’t go very far. I usually manually review the data samples before using it.

gdiamoson LLM-Deflate: Extracting LLMs into Datasets

What open source code do you use to pull synthetic data from LLMs?

gdiamoson Nvidia buys $5B in Intel

best news i've heard in days

gdiamoson Patrick Winston: How to Speak (2018) [video]

It looks like it was not included in this video.

Here is a link to a previous video - https://www.youtube.com/watch?v=Q7EuXNw0DaA&list=PL240A9CB42...

I think it is better if he explains it

gdiamoson Patrick Winston: How to Speak (2018) [video]

I had to rewatch his point about stage fright at the end several times before I finally got it

gdiamoson How is Ultrassembler so fast?

ptxas is a bit of a misnomer - it actually wraps the entire NVIDIA driver backend compiler

PTX isn’t the assembly language, it is a virtual ISA, so you need a full backend compiler with 10s to 100s of passes to get to machine code

gdiamoson The Theoretical Limitations of Embedding-Based Retrieval

Their idea is that capacity of even 4096-wide vectors limits their performance.

Sparse models like BM25 have a huge dimension and thus don’t suffer from this limit, but they don’t capture semantics and can’t follow instructions.

It seems like the holy grail is a sparse semantic model. I wonder how splade would do?

gdiamoson The Annotated Transformer (2022)

It's so good it is worth revisiting often

gdiamoson A message from Intel CEO Lip-Bu Tan to all company employees

Arm makes a specification and standard (the ARM ISA).

Apple licenses that and develops their own chip, which is then manufactured by TSMC.

So I guess if Intel dies the US will still have a few good CPU design firms, but no manufacturing

Also note that Foxconn (China) assembles the iPhones

Eg https://www.businessinsider.com/apple-iphone-factory-foxconn...

gdiamoson GPT-5

Scaling laws enabled an investment in capital and GPU R&D to deliver 10,000x faster training.

That took the wold from autocomplete to Claude and GPT.

Another 10,000x would do it again, but who has that kind of money or R&D breakthrough?

The way scaling laws work, 5,000x and 10,000x give a pretty similar result. So why is it surprising that competitors land in the same range? It seems hard enough to beat your competitor by 2x let alone 10,000x

gdiamoson Tokens are getting more expensive

The NVIDIA stock price keeps going up because accuracy/intelligence is more valuable than efficiency

Scaling laws let you spend more transistors and watts on intelligence

Do you want more tokens or smarter tokens?

gdiamoson Ozzy Osbourne has died

As someone who listened to the music, it’s surprising to see this as the top comment.

Yes their lyrics are dark. That was the point.

Eating animals isn’t what comes to mind for me. I also rationalize it that 100s of millions of animals are slaughtered every day, especially birds. Which one of those facts is darker?

It’s surprising to see what people are remembered for.

gdiamoson Felix Baumgartner, who jumped from stratosphere, dies in Italy

That’s only if you are planning to jump 100 more times.

gdiamoson Felix Baumgartner, who jumped from stratosphere, dies in Italy

probability is memoryless.

If you have been base jumping for 20 years, you have the same risk on your next jump as someone trying it for the first time.