Comment by nl | Hacker News Reader

Comment by nl | Hacker News Reader

Gemma-4-E4B-it scored 15/25 on my https://sql-benchmark.nicklothian.com/#all-data (agentic SQL generation).

The naming is a bit odd - E4B is "4.5B effective, 8B with embeddings", so despite the name it is probably best compared with the 8B/9B class models and is competitive with them.

Qwen3.5-9B also scores 15/25 in thinking mode for example. The best 9B model I've found is Qwen3.5-9B-Claude-4.6-Opus-Reasoning-Distilled-v2 which gets to 17/25

gemma-4-E2B (4bit quant) scored 12/25, but is really a 5B model. That's the same as NVIDIA-Nemotron-3-Nano-4B which is the best 4B model I've found (yes, better than Qwen 4B).

That's a great score for a small model.

>so despite the name it is probably best compared with the 8B/9B

It runs much faster than a standard 8B/9B model, the name is given by the fact that it uses per-layer embedding (PLE).

I love that you are doing this test. However, as it purports to be a test of "English-to-SQL", your hardest question (Q9) seems ungrammatical:

> Show order lines, revenue, units sold, revenue per unit (total revenue ÷ total units sold), average list price per product in the subcategory, gross profit, and margin percentage for each product subcategory.

In particular, the clause "in the subcategory, gross profit, and margin percentage for each product subcategory" is ambiguous, and I wonder if more models would pass if the English were reformulated to be correct.

(it's also notable that Claude Opus 4.6 and Sonnet 4.6 both "missed" this one)

Oh this page is great! I just released AIM [1] which is a tool that generates verified SQL migrations using LLMs, and I tested a bunch of models manually. I think I'll just link to your page too!

[1] https://github.com/alecthomas/aim

Very happy to see updates to your benchmark. Looking forward to inclusion of larger Gemma 4 models!

The medium one on OpenRouter didn't support tools when I tried it. I will update when there is one.

Reply on news.ycombinator.com