Comment by thewataccount | Hacker News Reader

Comment by thewataccount | Hacker News Reader

This is great news!

I don't mean to be negative but - has anyone at all found a use for any of the 7B models? And are you only using it because you're resource constrained? Or is it a purely academic thing for the moment?

This model is all over the place in the benchmarks, it looks roughly comparable to llama, and I know it's open to use unlike llama. But I haven't seen a model under 30B consistent enough to actually use for anything myself.

Hopefully the improved dataset will improve the next version, I truly want small models to work well, I'm sure it time it will improve.

EDIT: I mean this isn't even a trick question: "There are two shapes each a different color, one shape is red and one shape is blue. Shape A is red. Shape B must be what color?"

> This logic puzzle is based on a classic riddle that has been passed down for generations. The answer, of course, is that Shape B must be a red shape, as red is the only color that can be used to represent all reds.

> The key to solving the puzzle is to recognize that red can be used to describe color as well as blood, which is another word for shape.

What ????

EDIT 2: GPT3.5-turbo says "there's no definitive color for shape B"

Only GPT4 gets this right, "Since shape A is red, and the other shape is blue, shape B must be blue."

I have no idea how they struggle on this.

Definitely don't quantize to 4-bit at this param count, if that's what you're doing. The redpajama.cpp repo is a bit crazy for making that the default.

I'd say at this size don't count on accuracy. Think: fancy auto-complete, simple NLP task-doer, or a base for fine-tuning a task specific model. IME RedPJ-3B was too small to do anything at all other than ramble. The novel contribution of this model here being you can use this at work, and it has a GGML implementation unlike Falcon and MPT (last I checked) - meaning CPU-based inference on the cloud VM you use at work, maybe.

People that run 7B 4bit is probably because they don't have the memory for more. And there are tests showing that quantizing a bigger model is always better than running a smaller model full precision (i.e LlaMa 13B 4bit > 7B fp16)

> a base for fine-tuning a task specific model

Yes. The 7B size can perform very well when tuned on a specific menu of tasks:

1. Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Tasks

https://huggingface.co/papers/2305.14201

2. Gorilla: Large Language Model Connected with Massive APIs

https://arxiv.org/abs/2305.15334

Some of the most hilarity I've ever seen was asking various 7B models: "How old is a child who was born 9 years ago?"

Paragraphs and paragraphs of complex calculations, ending up with "9.5" or "11" or some other close-but-very-wrong answer.

> I don't mean to be negative but - has anyone at all found a use for any of the 7B models?

Productive/generative and logic-based tasks improve with the size of the corpus, but smaller models are useful for reductive tasks where florid prose and accuracy are not a requirement-- i.e. summarize this or classify that. I use dolly-3B for quick-and-dirty work like this since I can run it on spare hardware.

Commercially, you could probably get away with hot-swapping large models for small ones during periods of high demand-- for a chat service, kick every 3-4th exchange to a smaller, cheaper model to reduce load on the larger one. Nobody is perfectly coherent all the time...

Huh interesting it does appear pretty good at summarization when I tried it. Are they consistent enough to not modify the information or mix up numbers in your experience?

That's my main concern is it making "changes" to what it's summarizing for that usecase.

That's kind of expected since GPT2 was already relatively decent at summarizing.

Guanaco 7B quantized to 5-bit got this right on the first shot: "Based on the information given, we can conclude that Shape B is blue. Since Shape A is red, and the only other option is Blue, it must be Shape B."

WizardLM-30B-Uncensored (4bit)

> There are two shapes each a different color, one shape is red and one shape is blue. Shape A is red. Shape B must be what color?

> Shape B can only be the other available color which is blue since we know that shape A is red.

(I agree with you that <30B models are very weak)

Vicuna-13b (4bit) got the answer right, the first time as well.

Bard delivering on its first shot:

"Shape B must be blue.

The question states that there are two shapes, each a different color, and that one shape is red. This means that the other shape must be blue, since there are only two colors and one of them is already taken."

This is some Vizzini (The Princess Bride) grade logic

https://www.youtube.com/watch?v=rMz7JBRbmNo

GPT 3.5-turbo (via API) gets confused with this as well untill I tell it: "There are only two shapes."

GPT 3.5 Turbo gets the right answer for me every time.

GPT3.5 turbo almost got it for me, sort of did but

"If Shape A is red and Shape B is a different color, one possible color for Shape B could be blue. However, it's important to note that the color of Shape B could be any color other than red. Without further information or constraints, there is no definitive answer for the color of Shape B."

Not that it matters a ton here, I'm just surprised that 3.5turbo had issues with it at all.

First try on GPT 3.5 (through the chat web) I get

> Based on the information provided, Shape A is red and Shape B is the other shape. However, it is not explicitly stated what color Shape B is. Therefore, without further information, we cannot determine the color of Shape B.

I retried a second time and it answered the same

I retried a third time and it answered correctly

Reply on news.ycombinator.com