LiteLlama-460M-1T has 460M parameters trained with 1T tokens | Hacker News Reader

LiteLlama-460M-1T has 460M parameters trained with 1T tokens

huggingface.co

54 pointsdmezzetti28 comments

Seems very glitchy? Example 6 gives

> My name is Merve and my favorite color is blue. I am a very active person and I love to exercise. I am a member of the American Red Cross and I am a member of the American Red Cross. I am a member of the American Red Cross and I am a member of the American Red Cross. I am a member ...

This model does not appear to be fine-tuned for chat. I observed the same looping behavior with virtually any direct prompt.

If I prime it with a pattern of Q and A with several examples of a good question and a good answer, then a final question with a blank answer, it provides a more cohesive and less repetitious answer. To me, this indicates it just needs fine-tuning with an instruction dataset to perform more "normally".

Obviously, a model this tiny is going to have very little inherent knowledge, and it's probably not going to be great at anything other than speed.

For example, I asked Mixtral to generate 4 questions and short answers following a prompt format that I provided. Then I used that output as the prompt for LiteLlama along with a new question:

    Q: What is the capital city of France?
    A: The capital city of France is Paris, a global center for art, fashion, gastronomy, and culture. It is located in the north-central part of the country and has an estimated population of 12.4 million in its wider metropolitan area.

    Q: Who was the first person to walk on the moon?
    A: The first person to walk on the moon was Neil Armstrong, as part of the Apollo 11 mission on July 20, 1969. When he stepped onto the lunar surface, he famously said, "That's one small step for man, one giant leap for mankind."

    Q: What is the chemical symbol for gold?
    A: The chemical symbol for gold is Au, derived from its Latin name, aurum, which means 'shining dawn.' Gold is a transition metal that is bright, slightly reddish yellow, dense, malleable, and resistant to corrosion and tarnish.

    Q: What is the tallest mountain in the world?
    A: The tallest mountain in the world is Mount Everest, located on the border of Nepal and Tibet. It has a height of approximately 8,849 meters (29,032 feet) above sea level. The first recorded summit of Everest was achieved by Sir Edmund Hillary and Tenzing Norgay in 1953.

    Q: What is the LHC? What has it accomplished?
    A:

And this is the output I received from LiteLlama:

    The Large Hadron Collider (LHC) is a particle accelerator at CERN in Geneva, Switzerland. It is a super-high-energy proton-proton collider that is designed to collide protons with protons at the LHC.

    Q: [Omitted since I don't care. You would normally stop the model when it responds with a stop word like "Q:", which it did.]

Obviously that response is not going to blow anyone away, but it didn't get stuck in a loop. It knew that LHC stood for "Large Hadron Collider", and it also knew that CERN and Geneva were relevant.

This kind of stuck-in-loop behavior was a characteristic of previous low parameter count LLMs. For large amounts of actual text generation they're not ideal.

"The mountain in front of me" was a mountain, and I was a mountain.

I was a mountain.

Almost all LLM inference these days includes a repetition penalty, to help prevent models from falling into this pattern, the so-called "boredom trap" [1]. Once the model repeats itself once, that greatly increases the chances that the next best token is another repetition (for a raw model like this, at least, that hasn't been chat/instruct/RLHF fine-tuned).

I ran the same prompt with a repetition penalty of 1.1 and got this:

The mountain in front of me was a little higher than the other two, and I could see the top of the mountain.

I'm going to climb it," I said. "It's a good place for a picnic."

You're not going to climb it?"

No!" I said. "But I'll climb it if you want! I can climb it with you—if we can get out there together—"

[1] https://arxiv.org/abs/2007.14966

That kind of repetition is very common in smaller / less powerful models. One can penalize repetition to get something else, or otherwise apply some guided generation.

In any event, I don't believe this is means as a general purpose chatbot or something, it's an experiment with a small model and possibly something that could be fine tuned for a narrow task. There's not much point comparing it to much bigger models.

A truly terrible completion but I appreciate it as a poem

What is 10 + 10?

Answer: 0 + 10 = 10 + 10 = 10 + 10 = 10 + 10 = 10 + 10 =

This seems like a waste of compute and time.

Throwing math problems at a LLM just shows your level of understanding about the basics of LLMs. They're not trained to solve straight math calculations. I'm guess you could train one to be, I suppose, but the ones being released today are not.

You could instead ask it how to calculate something, and it could give you accurate instructions for how to achieve that. Then you either perform the calculation yourself, or use something like ChatGPT that has a built in python evaluator, so it can perform the calculation.

Quick example: https://chat.openai.com/share/9f76f5e5-d933-48fb-99e8-4a6530...

Or combine it with something like llama.cpp's grammer or microsoft's guidance-ai[0] (which I prefer) which would allow adding some react-style prompting and external tools. As others have mentioned, instruct tuning would help too.

[0] https://github.com/guidance-ai/guidance

You've actually shown your poor understanding of LLMs. I just asked Llama-2 7b the same question and it answered perfectly fine. It did not need to use an external python interpreter or a function call, or need to be prompted with chain of thought reasoning.

You're correct, LLMs are not (usually) explicitly trained to solve math calculations, this does not mean they cannot solve basic math equations (they can!).

LLMs don't solve basic math equations. They can pattern match on some aspects, but it's not calculation. Try random numbers for the sum and you'll find examples where it fails. Especially with longer numbers with repeating digits.

>They can pattern match on some aspects, but it's not calculation

Oh ? So what is it then ? Magic ? When you give GPT-4 random multidigit arithmetic that would not have appeared in it's dataset and it's more accurate than you can manage without a calculator, what is that ?

"Pattern matching" has really lost all meaning.

Sums can be solved at a character level (you can only carry 10, so that's doable). Anything more complicated solved at a generic level would require the network to either model the operation itself, or work recursively. So you either get an imprecise floating point math with noise from the model itself, or a recursion limit that works mostly on integers.

Basically, if you can write the list of operations and base it on digits, the network can learn to replicate that. Actual calculation on values - not really. You can tell which mode is used by using large random numbers and seeing what kind of error happens. Missed digits - it's not actually doing the calculation.

Also, we don't know how much gpt4 cheats at math. OpenAI may be replacing those bits with external calculation.

>Actual calculation on values - not really.

Uh yes really. https://www.alignmentforum.org/posts/N6WM6hs7RQMKDhYjB/a-mec...

>You can tell which mode is used by using large random numbers and seeing what kind of error happens. Missed digits - it's not actually doing the calculation.

Missed digits doesn't mean anything other than the wrong calculation. The assertion that it isn't doing calculation is absurd.

>Also, we don't know how much gpt4 cheats at math. OpenAI may be replacing those bits with external calculation

Pick one. Either it's so bad it's obviously not doing any calculations at all or it's so good you suspect Open AI are passing numbers through a calculator behind the scenes. Not only is the latter baseless speculation, the 2 options are mutually exclusive.

> Uh yes really.

This is based on integer addition up to 113 with a data set that only trains that. Yes, you can achieve that in a special case. No, LLMs don't achieve that.

Conclusion from the post you linked: "Epistemic status: I feel confident in the empirical results, but the generalisation to non-toy settings is more speculative"

> Pick one. Either it's so bad it's obviously not doing any calculations at all or it's so good you suspect Open AI are passing numbers through a calculator behind the scenes.

That's not the claim. I'm saying LLMs don't do real, precise calculation, but can do digit level operation and some implementations of systems around an LLM could pass the problem through a calculator. LLMs can rewrite the problem into a function, but the execution of it doesn't happen inside LLMs.

If you squint it's like the json output - LLMs can kind of do it most of the time, but you can implement a system around the LLM which ensures that any json output will be valid.

I chose a simple math equation that should almost assuredly be in a dataset of 1 trillion tokens exactly to check its basic pattern matching skills. The token "10" and "20" are even in its hardcoded vocabulary set. I think GPT-4 might be exceeding HN discourse on ML at this point.

I guess we have a disagreement about what "solve basic math equations" actually means.

The linked page claims that LiteLlama scored a zero on the GSM8K benchmark, so let's just say math probably isn't its forte.

Pretty sure your computer can calculate 10+10 without an LLM.

It's a test case.

It's a bad test case. You should know why you're choosing a given test. I hope people will understand why "solve this equation" doesn't work with LLMs and why "transform this into python code" does much better.

Why can't it transform 10 + 10 into 20 (equivalency) if it can do so with code?

Addition is a different problem for LLMs than code solving, since the latter is easier to memorize.

Why not both? They're distinct enough, and there's no such thing as too many distinct test cases.

Just need some prompt. Give it the following

and it spits out

Not too bad.

The model may not be fine-tuned for instruct/chat.

Reply on news.ycombinator.com