Hacker News Reader

Comment by simonw | Hacker News Reader

This article's credibility suffers a little from the way it talks about GPT-4o mini:

"just in front of GPT-4o-mini, which is, according to itself, a model with 1.3B or 1.5B or 1.7B parameters, depending on when you ask."

Then later:

"On the Artificial Analysis benchmark Scout achieved the same score as GPT 4o mini. A 109B model vs a 1.5B model (allegedly). This is ABYSMAL."

Asking models how many parameters they have doesn't make sense.

There is absolutely no way GPT-4o mini is 1.5B. I can run a 3B model on my iPhone, but it's a fraction of the utility of GPT-4o mini.

View on HN

It's strange that someone from FastML can be confused about this, unless it's supposed to be a bad joke.

This is just someone's personal blog/opinion. I wouldn't read too much into it... "The site is run by Zygmunt Zajc (pronounced “Ziontz”). ... An economist by education"

Ah, FastML is an extremely overloaded name.

I flagged it for these reasons as well. It's just a bad article. Shows very poor understanding of the basics of LLM workings, and the field in general.

Lingers on the "cheated" benchmark (lmsys) but never mentions all the other 3rd party benchmarks performed after the inference fixes, which are in line with what Meta originally published. To be clear, submitting a different fine-tuned model to one arena and releasing the untuned model without clearly mentioning this, is bad. But conflating the "human prefference" bench with the others and not mentioning the models capabilities on other benchmarks is also bad writing.

The MoE paragraphs are bad, and the writer never explains why the copy 17B vs VRAM size is bad, they just leave it there unexplained.

Poor form, I was expecting better from someone working in this field.

Correct, in that models know nothing about themselves other than what they are told. Deepseek R1 will tell you that it's created by OpenAI.

GPT-4o mini is supposed to be ~8b params from estimates.

The best source I've seen for the 8B number is this TechCrunch article: https://techcrunch.com/2024/07/18/openai-unveils-gpt-4o-mini...

> OpenAI would not disclose exactly how large GPT-4o mini is, but said it’s roughly in the same tier as other small AI models, such as Llama 3 8b, Claude Haiku and Gemini 1.5 Flash.

As far as I can tell all of the 8B rumors were seeded by that loosely sourced comparison to Llama 3 8B.

I know for a fact that Gemini 1.5 Flash is NOT an 8B model, because a separate model called "Gemini 1.5 Flash 8B" came out after that article was published - the first Gemini model with a documented parameter count. Flash 8B is priced at half the cost of regular Flash.

There's also this paper that mentions 8B but provides no source for that at all, which makes me suspect their initial source may have been that TechCrunch rumor: https://arxiv.org/pdf/2412.19260

8b params is a reasonable estimate.

As a sanity check, we can look at scores for how it performs. On livebench, GPT-4o-mini scores 37.63, right next to Gemini 1.5 Flash 8B at 34.37, and above Qwen2.5 7B Instruct Turbo/Gemma 3 4B at 29.22/30.13. And it's below Phi-4 14b at 40.68, and Gemma 3 12B at 41.25.

It's strange because there's no need to make this assumption about GPT-4o in order to demonstrate their point.

we are in for a lot of pain if seemingly intelligent people make mistakes like this. grabbing the number of params from what gpt gives you. how can you do that?

The comment at the top says it's a draft. It's not unreasonable to ask for random values from a GPT for "filler" for the draft (or even just make them up), just to stay in the flow, and then track down the real numbers later.

Reply on news.ycombinator.com