This is a solid improvement over Needle 2, which I tried using for a tool-calling interface to a Runescape database site. Unfortunately it's still not quite capable enough for my target compared to FunctionGemma.
Model Correct tool shape Exact arguments
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ━━━━━━━━━━━━━━━━━━━━ ━━━━━━━━━━━━━━━━━
FunctionGemma fine-tune, BF16 209/230 (90.9%) 196/230 (85.2%)
─────────────────────────────── ──────────────────── ─────────────────
Needle 3 fine-tuned W4A8 74/230 (32.2%) 47/230 (20.4%)
─────────────────────────────── ──────────────────── ─────────────────
Needle 2 fine-tuned W4 59/230 (25.7%) 43/230 (18.7%)