Hacker News Reader

Comment by justanotheratom | Hacker News Reader

This is nice, but it can get quite expensive.

Let's say I have a book and I want to ask multiple questions about it. Every query will pay the price of the book's text. It would be awesome if I could "index" the book once, i.e. pay for the context once, and then ask multiple questions.

View on HN

This more or less is already a thing and it's called RAG [1][2]. It essentially allows you to have a database of embeddings (in this case your book) from which a model can pull knowledge from while producing answers. As for the standard operation of these generative models, the context window is the only working memory it has and so it must see the entire text each time.

[1] https://arxiv.org/abs/2005.11401

[2] https://huggingface.co/docs/transformers/model_doc/rag

Cam you help me understand this? The research appears to be from a few years ago. Can this be used with Claude (for example)? How is it different to the approach many people are taking with vector stores and embeddings?

it's not different. RAG is a way to train embedding stores end to end

somehow got down voted on something I'm a professional expert at

Other people seem to be suggesting that the user would do the retrieval of the relevant parts of the book from a vectordb first, and then feed those sections along with the question as the prompt. Conceptually it is very similar (and it too uses vector database), but with RAG it would happen as part of the inferencing pipeline and therefore achieve better performance than the end user emulating it.

Yep, but your retrieval from the vector DB becomes your relevancy bottleneck.

With embeddings, you essentially can. Group the book into sections, embed each section, then when you do a prompt, add in the N most similar embedded sections to your prompt.

What if the question is "What are the main themes of this work?"

Or anything where the question answer isn't 'close' to the words used in the question?

How well does this work vs giving it the whole thing as a prompt?

I assume worse but I'm not sure how this approach compares to giving it the full thing in the prompt or splitting it into N sections and running on each and then summarizing.

You pretty much summed up the drawbacks of the embeddings approach. In my experience it's pretty hard to extract the relevant parts of text, especially when the text is uniform.

You could do multi level summaries etc but yeah this is all just band aids around token limits.

I don't think it's as much of a band-aid as it first appears since this roughly mimics how a human would do it.

The problem is that humans have continuous information retrieval and storage where the current crop of embedding systems are static and mostly one shot.

Humans have limited working memory, they quickly forget short term memory (unless it's super significant) and our long term memory fades selectively if not reactivated or significant (intense).

This weird leaky memory has advantages and disadvantages. Forgetting is useful, it removes garbage.

Machine models could vary the balance of temporal types, drop out Etc. We may get some weird behavior.

I would guess we will see many innovations in how memory is stored in systems like these.

That is solved by hypothetical embeddings.

Background: https://summarity.com/hyde

Demo: https://youtu.be/elNrRU12xRc?t=1550 (or try it on findsight.ai and compare results of the "answer" vs the "state" filter)

For even deeper retrieval consider late interaction models such as ColBERT

Any material comparing the different embedding models? I'm working on information retrieval from government documents and without any ML experience it's daunting

I'm not understanding how that works compared to having the full text?

Does the embedding structure somehow expose the themes? And if so, is it more the embeddings that are answering the question by how it groups things?

The real gain would be if we were able to use the 100K Context Windows and not this "embeddings trick". The embeddings work only in some cases where the answer is in a short part(s) of the document. If user asks something like "what are the main ideas?" or "Summarize the document." or any question that needs context from large portions of the book/pdf/file, then it will not work with the embeddings trick that use just short passages in prompt. But if large context windows costs are high, we need to keep using embeddings and few text parts.

The price on this will plummet over the next few years, the economic benefits are too large

Yes, but physics trumps economics.

The economic benefits of mining asteroids are also too large to ignore yet here we are, levelling villages to dig for coal.

Just a few manufacturers hold the effective cartel monopoly on LLM acceleration and you best bet they will charge out the ass for it.

I'm wondering what level you're thinking. Cloud vendors? GPU vendors? Fabs?

Given what's used right now to my knowledge, the main ones would be Nvidia's tensor cores, Apple's M chips and Google's cloud TPUs. All of that's TSMC I think?

Market competition and innovation in both ML and hardware has consistently driven down the price of AI in the past decade. You only have to look at where we are with capabilities today compared to ten years ago when CIFAR100 classifiers were the state of the art.

Barring a Chinese invasion of Taiwan, these APIs will halve in price over the next year.

Well here's to hoping I guess.

What would a Chinese invasion of Taiwan do to "tech", it sounds like it would be awfully catastrophic ?

On the other hand, for all peoples worrying about China, they are pretty restrained given the enormous turmoil the invasion would cause. If they wanted to break the USA, now would probably be the time to do it?

Well for one thing the US seems committed to blow up TSMC to prevent China getting the tech.

https://www.theregister.com/2023/03/14/us_china_tsmc_taiwan/

Planning for the inevitable?

> The economic benefits of mining asteroids are also too large

Ironically, that's an example I like to list as "pure sci-fi fantasy, divorced from economic reality."

The total cost of the iron ore that goes into making a new a car is about $200-$300 dollars, depending on various factors (size of the car, ore spot price, etc...).

Even if -- magically -- asteroid mining made not just "iron ore", but specifically the steel alloy used for car bodies literally free, new cars costing $30,000 would now cost... $29,700.

You can save more by skipping the optional coffee cup warmer, or whatever.

In reality: 90% of iron and steel is recycled, and asteroid mining is not magic.

Why would anyone want to mine asteroids for iron, given that it's the most abundant element on Earth? I don't think I ever recall seeing a proposal like that outside of the broader notion of "space factories" (where such mining makes sense to reduce the cost of shipping materials back and forth, not because it's cheaper as such).

It's stuff like platinum and germanium that makes asteroid mining potentially interesting.

Metallic asteroids are mostly nickel-iron.

On Earth, geological processes concentrate elements into ores, primarily through volcanic and hydrological means. Neither are available in small, cold asteroids devoid of liquid water. Hence asteroids are generally undifferentiated mineralogically, making mining them much less economically viable.

You often see total quantities listed as an amazing thing, glossing over the fact that the Earth has more of everything and in usefully concentrated lumps.

Not sure about this one but you can usually ask multiple questions in one shot at least

Generation is more expensive than the prompt input (for Claude v1, generation is 3x the cost; for GPT-4 it's 2x the cost)

It makes the economics slightly trickier.

I wonder why this is? Naively there's no difference between the two from a transformer standpoint.

Perhaps it's because under the hood there's additional safety analysis/candidate generate that is resource intensive?

Well each additional token generated requires rerunning the model right to find the next likely token given the previous one

Naively, yes, but you can cache the bulk of that "rerunning" [1]. That said the (non-flash) attention costs go up with the length of the sequence so perhaps this is just a simpler way to approximate these costs.

[1] https://kipp.ly/blog/transformer-inference-arithmetic/

Normally the inputs are padded out to the context length [1] and so the cost to embed 1 token or N tokens is the same. The output is produced token-by-token and so the amount of GPU time increases with the number of output tokens.

[1] I'm not sure if these huge context lengths are achieved the same way (i.e. a single input vector of length N) but given the cost is constant for input I would assume the resource usage is too.

This doesn't match my mental model (or implemented model in the case of GPT2) of how self-attention works (you need to calculate the residual stream for each individual token, attending to all prior tokens before it). Have a link?

I work on infrastructure for serving large language models but I don't have any background in ML, so my perspective is looking at these models as a black box (and also conversations with the people that do the ML stuff). It is the case in practice at least from a latency side that with a fixed context length N, embedding any number of tokens from 0 to N takes the same amount of time. Perhaps it's a difference between the conceptual and actual implementation on GPU?

edit - This occurred to me after the fact but I wonder if the difference is that the use case I work with is processing batches of many different embedding requests (but computed in one batch), therefore it has to process `min(longest embedding, N)` tokens so any individual request in theory has no difference. This would also be the case for Anthropic however.

Ah, you're thinking about embeddings which are basically the encoder stack on a traditional transformer architecture. Modern GPT-like models (including Claude), however, drop the encoder and use decoder-only architectures.

I could imagine something where encoders pad up to the context length because causal masking doesn't apply and the self attention has learned to look across the whole context-window.

Decoder only architecture? What is this? That doesn't sound like a transformer at all, are you saying gpt4 uses a totally different algorithm?

Nope, a decoder only transformer is a variant of the original architecture proposed by Google [1]. All variants of GPT that we know about (1 through 3) all roughly use this same architecture which takes only the decoder stack from the original Google paper and drops the encoder [2]

[1] Original Google Paper - https://arxiv.org/abs/1706.03762

[2] Original GPT Paper - https://s3-us-west-2.amazonaws.com/openai-assets/research-co...

How can it work without an encoder?

Everyone serious batches together short prompts so the cost is roughly proportional to the tokens.

It's because the input tokens can be batch-processed in a single forward pass through the model, while generating tokens requires one forward pass through the model per token.

If you do the math of how much memory bandwidth is required by a forward pass vs. how much compute, you'll see that inference is entirely limited by memory bandwidth and will use compute resources very inefficiently. In contrast, input processing is able to fully use the available compute.

Of course, there are ways to mitigate this problem, like processing multiple token streams in parallel, but the fundamental problem remains.

Ah, this makes total sense. I was thinking about FLOPs in the abstract and not about the wall-clock time. Thanks for the explanation.

Yes, caching the states of the sequence would make sense. An issue is that it's still more expensive to compute the new tokens even if you cache the states viewed so far

The analogy I can think of here is a pointer, but AFAIK the context would always need to go along with the prompt unless you could tweak internal state to bias towards the context.

Otherwise, it might make sense to have a separate routine which compresses the context as efficiently as possible. Auto encoder?

Reply on news.ycombinator.com