How long can open-source LLMs truly promise on context length?
lmsys.org
lmsys.org
Falcon 40BN, after extensive use, is the only model that has crossed the "I'd build a product on this" threshold in open source. I expect my tune won't change there until we see larger Open Source models trained on high quality data. These are the models that will benefit from the extended context lengths.
But longer context without a model capable of utilizing the context to actually do things, is yeah, unsurprisingly, not useful.
There's also not really an incentive to currently make that happen in Open Source. If you can train a big model, with a unique high quality dataset you've assembled, using some novel techniques on some performant serving infrastructure, you basically have a blank check right now.
Edit: On my last point, I'm going to say I was being very jaded. I think the community is doing a lot of awesome things and I know of a few efforts personally to train these larger models for open source consumption. I don't mean to be so overly negative here.
If you're looking for an incentive to make this happen in open source, how about the team behind MPT-30B just getting acquired for $1.3 billion.
There are a LOT of well-funded groups trying to make an impact in open source models right now.
If you look at Mosaic, their value, their secret sauce, is the infrastructure. The model, to say it cynically, was marketing. Now it's true it can be marketing and help push open source forward at the same time.
More of a separate point, but I also think models are going to be fairly quickly commoditized on a couple years timescale, so the infrastructure to train a model or serve a model, or the thing built on the model is the value, but right now models are still the value because they're so new and we haven't finished (in fact we're really only starting) the race to the bottom.
I would be very surprised if the closed source research teams can maintain their advantage over the next 12-24 months - especially given the number of enormous GPU clusters that are becoming available now.
High quality "open" data is an tricky problem, but the community has already come up with some impressive datasets for finetunes, like the roleplaying collection used for Chronos and Pygmalion. I think this is going to get better too, epecially if cheap models can "distill" lower quality human datasets.
I agree and am eagerly awaiting this on the hardware side and trying to participate in this where I can on the software side.
A research think tank subsidized by UAE’s oil wealth. Somebody get Masayoshi on the line, the Saudis must be hungry for deals for their vision fund.
People who get those grants are proud of the fact and like the money.
For, say, continuing a story, even a small LLM will benefit from a mega context even if it struggles to retrieve specific information... As long as perplexity isnt hit too much, which depends on the method.
But this is more of an issue for summarization or information retrieval.
We have the SuperHOT LORA and the associated paper, the Salesforce model, MPT, and this. Any other long context news I am missing?
Here you go, https://www.reddit.com/r/LocalLLaMA/comments/14lz7j5/ntkawar...
This is so exciting because contacts length has been a problem with these models for so long and it's awesome to see open source finally cracking that egg.
I need another GPU.
edit:
I'm actually a little bit skeptical of this. Yes it's dynamically scaling which is great when you have a model that's not fine tuned, but I think it's not going to work out too well when you try fine tune a model on a target that's moving like that. I'd rather one that stay static so that perplexity is always increasing up to the max rather than doing much those graph shows where it gets worse over time.
That said I don't really know what I'm talking about so maybe it'll be better regardless.
There's a symbiotic relationship between the open source community and the academic research here: the broader community can explore a ton of different improvements, at a much more rapid speed than either the academic research (slower because it has additional objectives) or closed-source (because the lack of sharing means that a lot of low-hanging fruit gets overlooked). Academic research can build the bigger, more experimental projects that create new capabilities. Research can also do the testing and research that the broader dev community doesn't have the time and resources for, giving us a better idea of what parts actually work best and why it works. It can be very valuable to have a paper that tells us something everybody knows, because you can verify the common assumptions empirically and give us numbers that tell us things like how well it works.
I expect to see a lot more discoveries that bounce back and forth between the audiences, because both groups benefit in different ways.
LoRA: https://news.ycombinator.com/item?id=35288015
The Salesforce Model (XGen): https://news.ycombinator.com/item?id=36514936
MPT (Mosaic): https://www.mosaicml.com/blog/mpt-30b
https://news.ycombinator.com/item?id=36533816
The lmsys team is already looking into it, expect further progress soon.
It's really amazing to follow this progress in realtime, even if the pace can sometimes feel exhausting!
Yes, it actually works. That's what matters at the end of the day.
But no, eventually you're going to have to fine tune on something that has a larger context or training a brand new model with the position embeddings semi randomized so that it learns to generalize in the first place instead of needing hacks like this to function.
But training a model costs millions and millions of dollars, so we're going to have to wait until some very generous group decides to do all that training and then release that model open source.
Or releases the model file for a paid charge fee, I'd pay like $200 for a good one.
It's like being in conversation with an Alzheimer's patient, who keeps forgetting the last sentences?
So, open source models don't do well, while closed source models do?
And, this post documents a way to better measure that important metric?
When you have a conversation with one, you are actually feeding it the entire conversation every time you reply, usually with specific formatting the model is trained to "recognize" as a ongoing conversation, like its continuing a long story. There are some other formatting details, like the initial "instruction" always being preserved at the top as the old conversation gets truncated, and there is some caching done so it doesn't have to encode the entire conversation every time.
The context size is how much conversation you can feed them at once.
The popular open models were essentially limited to ~2K tokens until recently.
It's worth noting that "soft-prompts" were a thing that the coomer crowd was already using with openassistant. Not sure why the mainstream NLP crowd is being so damn slow in implementing stuff like this
NLP is shocking undertooled, and I wrote a whole gist complaining about this that got to the front page of HN awhile ago: https://gist.github.com/Hellisotherpeople/45c619ee22aac6865c...
But the text token merging people do for negative prompts is fast iirc.
Not that you are wrong. LLM prompting does seem quite primitive compared to zoo of encoding/prompting methods we have for SD... And some of it is directly applicable to LLMs. I never even thought about that before.
Textual Inversion/Soft Prompts are not made "obsolete" at all by Loras. Textual Inversion is far faster to train, extremely effective on newer models like SDXL which have better clip encoding, and has neat properties of not messing with the general weights of the rest of the model, which is sometimes desirable.
You can modify it to run locally on a GPU, though it is a bit slow to train. It provides different advantages when compared to a LoRA.
As in, I want to be able to feed the LLM arbitrary embedding vectors - whether they come from averaging embeddings of different prompts, some general optimization pattern (textual inversion style, indeed), or just iteratively shifting the input vector in some direction, walking down the latent space and generating output every step, just to see what happens.
I ask that, and I get blank stares, or non-answers like "oh, it's always been available, you can e.g. put text in, get embeddings out". Yeah, but I want to do the reverse. It's like I was making so big a category error, asking such nonsensical question, that no one can even parse it. But am I?
The way a transformer-based LLM works is that right after tokenization you get a "vector of embedding vectors" i.e. an embedding for each token, and that gets processed by the actual transformer layers. If instead of a list of tokens you want to pass in a list of embedding vectors, that is trivial (assuming you're running a LLM with your own code locally), that would be a <5 line function.
However, I'm not really sure when/why I would want to do that, as "averaging embeddings of different prompts, some general optimization pattern (textual inversion style, indeed), or just iteratively shifting the input vector in some direction, walking down the latent space and generating output every step" are NOT really enabled by that - since that inherently must be a vector of token-embedding vectors combined with positional embeddings, you can't just average different prompts, and there isn't a clear way how to tweak the individual token embeddings that would map to tweaking the whole prompt in some meaningful direction.
In essence, your question seems to make sense if the text was converted to a single "embedding vector", but it's not, at every layer of a transformer model the state is a list of separate embeddings distinctly separated at a token level.
That sounds like what I wanted. I haven't got to the point of running open-source LLMs locally yet, so I didn't realize it's as easy as you say - rather, since I haven't seen it being mentioned as something you do with LLM, I assumed it must be nontrivial.
> However, I'm not really sure when/why I would want to do that
High-level dream: I want to explore the latent space of an LLM. Get a feel for how it's organized, and what of it is spurious, and what structures seem to be common among LLMs. Find answers to questions that will come to me when I "go there" and "look around".
It might be a futile pursuit - but I feel it's not the idea that's the problem, but rather that high-dimensional spaces might be too big to explore without mathematical understanding that's way beyond mine.
> as "[the specific ideas I listed]" are NOT really enabled by that - since that inherently must be a vector of token-embedding vectors combined with positional embeddings, you can't just average different prompts,
In general, or at all?
Am I wrong in suspecting that averaging the embeddings of "orange" and "apple" and "banana" may give us a new embedding that, fed to the LLM, will make it output something meaningful (vs. random)? From what I read today, elsewhere in the thread, it seems that this works as expected for Stable Diffusion family of models.
And sure, if I take two random multi-token prompts, I expect the result will make LLM output some noise in the shape of word salad. But what if it's multiple closely-related prompts? I have my suspicions, but I want to find out for sure.
> and there isn't a clear way how to tweak the individual token embeddings that would map to tweaking the whole prompt in some meaningful direction.
Shifting prompts in some meaningful direction may not be trivial (or useful), but it's not the only interesting thing to do. What if I push a single token in the prompt a little bit in some direction? What if I push all of them? How things change with subsequent pushes?
> In essence, your question seems to make sense if the text was converted to a single "embedding vector", but it's not, at every layer of a transformer model the state is a list of separate embeddings distinctly separated at a token level.
I'm definitely not assuming the text can be converted to a single "embedding vector". My questions boil down to, where is debug-level access to the LLM? Why is everyone focused on stringing tokens, instead of looking at what's lurking between the tokens and their sequences, in areas of the latent space you can't easily get to without embeddings that don't map to any possible prompt.
IDK, maybe my mental model of transformer models is completely broken, or maybe all those questions have been asked, and didn't lead to anything of interest, and I just can't seem to find the right papers covering this.
Many have interpreted recent advances in LLM as a sign that strong AI is just around the corner. Yet, their static nature is an obvious blocker to ever achieving that – humans, even animals, learn continuously, unlike LLMs where training and runtime are two completely separate processes.
Which makes me wonder – how difficult would it be to have something like an LLM which continuously learnt just like humans do? Could it be done by modifying existing LLM architectures such as Transformers, or would it require a completely different architecture?
The math works the same either way. Keep in mind... Llms learn orders of magnitude faster than humans due to parallel nature of transformers.
Whereas, in biological brains, the weights are updated continuously.
Neuroscientists/psychologists/etc believe [0] humans have four tiers of memory: sensory memory (stores what you are experiencing right now, lasts for less than a second); working memory (lasts up to 30 seconds); intermediate-term memory (lasts 2-3 hours); long-term memory (anything from 30 minutes ago until the end of your life).
We don't need to sleep to form new long-term memories – if at dinner time you can still remember what you ate for breakfast (I usually can if I think about it), that's your long-term memory at work. What we need sleep for, is pruning our long-term memory – each night the brain basically runs a compression process, deciding which long-term memories to keep and which to throw away (forget), and how much detail to keep for each memory.
Regarding your juggling example – most neuroscientists believe that the brain stores different types of memories differently. How to perform a task is a procedural memory, and new or improved motor skills such as juggling are a particular form of procedural memory. How the brain processes them is likely quite different from how it processes episodic memories (events of your life) or semantic memories (facts, general knowledge, etc). Sleep may play a somewhat different role for each different memory type, so what's true for learning juggling may not be true for learning facts.
Recurrent neural networks 'learn' continuously by changing their weights (In effect) based upon the 'previous state'. There are papers showing that attention mechanisms in transformers basically provide a 'weight' update function so that models can 'learn' to accomplish a task based on a few examples. In other words, transformer networks 'learn' to train themselves based on the examples given. It is not so beyond beliefs that recurrent neural networks do the same thing. They learn to set up their internal states such that future tasks can be affected by previous input and patterns. In fact... playing around with models like RWKV, you soon learn that this is a fundamental part of the model. If you start talking to it in a certain way, it will start echoing that back. Clearly it's 'learning'.
That seems closer to what we do, we recall the important bits without every single word said.
But its also several passes through the LLM instead of one, and is prone to distortion.