Hacker News Reader

Comment by dmezzetti | Hacker News Reader

It's still up for debate as to whether long context windows are worth it. It's also not a cheap way to solve a problem.

Clearly hardware vendors love LLMs. But it's just a highly inefficient approach for a lot of problems.

Having them available is still hugely beneficial, even if they end up too expensive to use in most production use-cases.

They would still be valuable for prototyping, where fast iteration makes it possible to learn more about the problem you are solving and whether it is even worth solving. They are also valuable for iteration-and-distillation approaches, where you can use the data generated from an expensive model to train a cheaper model.

It seems like a very long input to an LLM can act as additional training data. A possible future is that LLMs will be used to bootstrap AIs that will be "trained" by feeding them a giant prompt at the start. Might end the era of "supervised" learning, "reinforcement" learning, numerical methods and gradient descent -- those techniques are awkward and procrustean. Imagine what you could do if you could improve a neural network - maybe to AGI level - just by talking to it in English?

So it looks like a VERY good idea. Who gets the gist of what I'm saying?

> Imagine what you could do if you could improve a neural network - maybe to AGI level - just by talking to it in English?

You'd get a repeat of Microsoft Tay, the short-lived 2016 chatbot that had to be taken down after not even a full day because 4chan managed to turn it into a full-blown hate spreader in hours [1].

Before something like this can be reasonably released to the Internet at large, we need to find out how to teach the "ingestion" part how to judge the input that it's being presented with... basically, AI school. The same way we teach our children that it's not OK to steal other people's stuff, to not be the one first throwing punches or to not discriminate against other people, or that there are reasonably trustworthy media and absolutely untrustworthy, we need to teach AIs. And we'd also need to figure out ways to teach an AI basic, hard truths: the Holocaust happened, the Earth is a globe not a pizza, the Earth is not hollow, and the moon landings were real.

At the moment, we're half-ass attempting that by annotated training data (and this is the true moat of OpenAI, not the weights or prompts!), but even as we invest literally millions of hours of training compute time, an average high schooler that has been trained for 18 years can reasonably pass the above-mentioned criteria to be a productive member of society.

[1] https://en.wikipedia.org/wiki/Tay_(chatbot)

> It's still up for debate as to whether long context windows are worth it.

As someone who's run into problems related to short context windows quite often, can you explain what "worth it" means? Also, when you say "not cheap", what alternative do you have in mind?

Continuous fine tuning so you don't have to pass everything in the context with every query would definitely yield a better UX.

Afaik the few-shots learning abilities of LLMs are much better than what you can achieve with fine tuning when you only have tiny samples.

Of course it is true for real long context but it's not clear if its going to work with the sparse context hacks intended to keep memory size low.

Having the ability to keep the same coding session forever without having the LLM forgetting all the time and making the same mistakes over and over would be a game changer.

I'd argue the fine-tuned UX is often worse.

People like RAG-based solutions because you can include references in your answer very easily (eg, Perplexity, or see the DAnswer "internal search" product launched today on HN). That is extremely hard to make work reliably from a fine-tuned model.

I mean UX from the perspective of the person doing the search, not the engineer. I don't dispute that fine tuning is harder to implement.

I mean from the perspective of the person doing the search too.

They often want to see the reference - for example the LLM is constructing an answer about some company policy question it is good to include references to the actual company policy documents by URL. This is easy using RAG, but very hard to do reliably just using a fine tuned LLM.

See Perplexity for a great example of this at web scale.

This is true. If LLMs recalled perfectly their purveyors would get into legal trouble :)

Retrieval Augmented Generation aka RAG with a relevant context. Long contexts continue to have the lost in the middle syndrome.

Gemini doesn't have the "lost in the middle syndrome"

We don't have enough data on that yet to know for sure. It was just released this week.

But nonetheless, sending 2M tokens to a LLM isn't an efficient way to solve most problems.

LLMs in general are likely to be a very inefficient way to solve many problems they're currently being used for. But so long as we don't have something better, brute force remains a valid approach.

> We don't have enough data on that yet to know for sure. It was just released this week.

I think there is sufficient evidence to think it works. A bunch of people who do have access have put it through a bunch of real-world exercises to test this, and while GPT4 128K and Claude 200K both fail Gemini is passing.

I agree sending 2M tokens to a LLM is costly at the moment and that RAG has a bunch of advantages in some circumstances.

A cheap way to solve a problem has always been throwing more compute power at it.

Figuring out new algorithms is monumentally more difficult than more hardware, and even then when more efficient algorithms are found, we throw more hardware at it and get a million times more done.

Billions of dollars spent on hardware kind of makes this rational to try

It depends what problem you're solving. If it's a high frequency request, like a chat response, it's far too inefficient. Most web APIs would consider it bad practice to read 2MB of data on every request, even worse when you consider all the LLM computation. Instead, use RAG and pull targeted info out of some sort of low-latency database.

However, caching might be a sweet spot for these multi-modal and large context LLMs. Take a bunch of documents and perform reasoning tasks to distill the knowledge down into something like a knowledge graph, to be used in RAG.

The cool thing about this paper is, actually it is a (relatively) cheap way to solve this particular problem (LLM on 2048k window), because they are using pre-trained models like LLaMA2 and Mistral and extending them to 2048k windows using a novel technique, rather than training a model from scratch with 2048k tokens which would be prohibitively expensive to mere mortals.

Reply on news.ycombinator.com