XGen-7B, a new 7B foundational model trained on up to 8K length for 1.5T tokens

It seems to use a different tokenizer than LLaMA, though the neural network architecture is the same.

There's also OpenLLaMA, which has a 13B version as well and is a straight drop-in (except for code generation due to multiple space tokenization: https://github.com/openlm-research/open_llama#update-0615202...).

XGen-7B is probably the superior 7B model, it's trained on more tokens and a longer default sequence length (although both presumably can adopt SuperHOT (Position Interpolation) to extend context), but larger models still probably perform better on an absolute basis.

The main use case is that it's probably the only size consumers can run on their personal devices. If you don't want your data going into an external platform like OpenAI it's the only solution even if it's not very usuable.

> What use cases do people have for these smaller LLM's?

None. Training a functionally useless model and releasing it is a great way to demonstrate that your company is hip and current. That way when prospective clients ask about AI you can vaguely gesture at some model that you released and say you employ cutting edge AI experts.

Bert had lots of use cases and this one is supposedly stronger model.

They're only modestly worse than text-davinci-003 in my experience, and you can finetune them cheaply to do simple tasks e.g. triage human input and decide where to send it. But yeah, if you can afford to run a larger model or pay OpenAI for gpt4 calls, that's gonna work a lot better.

If you're relying on prompting for the 7B models IMO you're gonna have a bad time — they're mostly toys at that: interesting output but not consistently useful. But finetuning gets better results, and it's cheap to finetune.

7B LLaMA is a terrible general purpose model, but the finetunes are pretty good at very specific roles, like dialogue/roleplay, a dungeon master bot or even code completion.

The metrics are good though, perhaps placing this closer to 13B.

And 8K context is huge. When you can stuff that much example text in, it gives the model more to "latch onto," and its also the point where you would start worrying about RAM/VRAM consumption for a ~13B model.

The use case is people who got some fear of missing out on AI and also they are not capable of seeing the difference between the model outputs themselves. Maybe you will say I'm elitist but it's disturbing to me how many people literally can't tell the difference between gpt 3 and gpt 4 outputs, it's like the "It's the same picture" meme. Maybe that makes me a hipster gpt connoisseur elitist that I can recognize and have opinions about those differences between versions and I hate using the small ones and I think only the very biggest ones are good.

I did some old school NLP before, but don’t really work in the field anymore. As a generative model. As a general purpose generative model, maybe this isn’t very useful. As a foundation to make models to perform text classification and information extraction tasks, this could be very useful. For these kinds of tasks, you can still get good results with the classic bag of words type approaches people were using 30 years ago even. I remember when transformers first came out, limitations in sequence size made them unusable for some classification tasks.

I feel like this would work well as a local LLM for a home assistant type setup. Fully local instead of having Alexa send everything to the cloud.

I've had very good results with even a 3.5 B model (Fastchat-T5) for retrieval augmented generation (aka putting information into the context window and letting the model rephrase it).

1. The trained model has 7B parameters or weights for each neuron.

2. It can handle upto 8k tokens. Tokens are usually some representation for a word. If your tokens are characters then, "h", "e", "y" represent 3 tokens for hey. Most of the algos use byte pair encoding. For example "hand-le" has two tokens "hand" and "le". This is a very crud example which is enough to give the gist but is not accurate. You can look into byte pair encoding for more details.

3. The token size 1.5T token means they have huge variations for input and output. Simply put, it was trained on large data corpus.

I hope this simplifies it. You can research further if you are interested! Hope it helps!

A (overly) simplified explanation:

- 7B means 7 billions parameters.

- 8K length means the size of input/output is 8K tokens.

- 1.5T tokens mean the training set has 1.5T tokens.

A: What's a parameter?

Q: More parameters your model has, more complex relationship it can represent. For example let's say you have a function f(x). This is a 2-parameter model:

f(x) = ax + b

This is a 4 parameter model:

f(x) = ax^3 + bx^2 + cx + d

As you can see as the number of parameters grows, the function is able to represent more complex relationship between f(x) and x.

A: What's a token?

Token is a way to encode text, like ASCII or Unicode. Unlike Unicode, tokenizor usually favors common combinations of alphabets. For example, "the" is a single token for GPT-3 tokenizor, but "eht" is two tokens (e and ht).

* Note that the number of parameters is more like an "upper limit" of the model's capabilities. If your a, b, c, d are just random shit, it's still a 4-parameter model, but it's still useless. The whole concept of "training" is just "finding the best parameters".

Here are some high level answers:

"7B" refers to the number of parameters or weights for a model. For a specific model, the versions with more parameters take more compute power to train and perform better.

A foundational model is the part of a ML model that is "pretrained" on a massive data set (and usually is the bulk of the compute cost). This is usually considered the "raw" model after which it is fine-tuned for specific tasks (turned into a chatbot).

"8K length" refers to the Context Window length (in tokens). This is basically an LLM's short term memory - you can think of it as its attention span and what it can generate reasonable output for.

"1.5T tokens" refers to the size of the corpus of the training set.

In general Wikipedia (or I suppose ChatGPT 4/Bing Chat with Web Browsing) is a decent enough place to start reading/asking basic questions. I'd recommend starting here: https://en.wikipedia.org/wiki/Large_language_model and finding the related concepts.

For those going deeper, there are lot of general resources lists like https://github.com/Hannibal046/Awesome-LLM or https://github.com/Mooler0410/LLMsPracticalGuide or one I like, https://sebastianraschka.com/blog/2023/llm-reading-list.html (there are a bajillion of these and you'll find more once you get a grasp on the terms you want to surf for). Almost everything is published on arXiv, and most is fairly readable even as a layman.

For non-ML programmers looking to get up to speed, I feel like Karpathy's Zero to Hero/nanoGPT or Jay Mody's picoGPT https://jaykmody.com/blog/gpt-from-scratch/ are alternative/maybe a better way to understand the basic concepts on a practical level.

1) 7B foundational model means this is a base model that has not been fine-tuned on the "prompt: response:" (instruct) structure, and it has 7 billion weights/biases. It's the working size of the model.

2) Currently every model that can run locally was trained with a 2K context size. It's a hard limit on prompt length. There have been recent advances with [A] position interpolation, but those methods explore fine-tuning/loras. This base model was trained with 8k sequences.

3. 1.5T tokens is the size of the total training corpus. Training cost and time increases with training size. [B]

A. https://arxiv.org/abs/2306.15595

B. https://www.semianalysis.com/p/the-ai-brick-wall-a-practical... (Jan 2023)

Why not ask the AIs themselves? They are pretty good at explaining this type of thing.

It's an open source language model with seven billion parameters (a measure of its complexity), and a longer than typical sequence length of 8K, which allows the provision of more context when querying the model. For example, it allows you to better generate text in someone else's voice by providing a longer example of their work.

https://en.wikipedia.org/wiki/Foundation_models

I'd say for research purposes. HackerNews seems to tend to mostly represent the LLM consumer viewpoint, but these waves of models being released are honestly more interesting from a research than a user point of view. As a LLM user, you're (vast generalization/simplification) really just interested in the best of N models, but as a researcher, I'm super interested in each model's performance and analyzing the reasons for any differences.

With this model (and they say this in the blog post), they were testing the hypothesis that training on a longer context size would provide more performance at the same parameter count/inference FLOPs. From a quick perusal of their post, it looks like this was true, and we should train all future models with as long of a context size as we can afford.

Long way from that

If you’re curious, you can check the progress of many open source LLM’s and how they perform on various evals here:

https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...

Nobody doing xB models is participating in any AI races, at this point those are useless toys with garbage output.

We’re still in the early innings.

I don't think we need anymore high-flying search engines, do we?

salesforce air esearch ;)

I think we are only at the beginning. Call it first generation. LLaMA was even missing many improvements that existed before it (xPos, Multi-Query Attention, Blockwise Parallel Transformer)

Many researchers are improving very fast, and I would bet that soon we will see more efficient LLMs.

When it's superhuman.

To quote Dennis Reynolds, it hasn't even begun to peak.

In terms of Gartner hype cycle, we are at the peak of inflated expectations for llms I reckon.