HenryNdubuaku

Yes, it was really difficult to compress meaningfully intelligence down to that, and there are many limitations we are aware off and still improving on.

100%, data was honestly most of the work, Needle 3 is trained on 360B tokens of structured data and we spend way more time on the generation pipeline than on the model. On Led Zeppelin, Needle doesn't actually need to know it, arguments are copied from the request so it just lifts the name into the artist field. The knowledge went into the engram btw, 70M of the 121M params are n-gram tables, so it can tell artist vs song without an MLP. Also yes, "play their second album" won't work, that needs the world knowledge it doesn't have.

Makes a lot of sense! declare one record with the ~20 keys as fields, cuisine and friends as enums with the common values, and the grammar can't produce a value outside the set; fields with no evidence come back empty, so the output is exactly the diff.

For "which place", query nearby POIs from location in the app and pass the candidate names as an enum field, so Needle picks rather than guesses.

Two caveats: it's text-in, so you need on-device STT first, and opening_hours syntax is the risky bit, so either put the format in the description or capture the raw hours and normalise in code.

presets updated for you now :)

True! We finished Needle 3 before Jev launched. Also, we are merely chasing one DeepSeek v4 Flash capacity with a small model, DeepSeek models are really good.

thanks for these haha, you can actually edit the tools and/or their descriptions, the demo is just a "get started" preset. But still we do have room for reasoning improvement!

Fair, we gotta do a better job at explaining this properly!

So an 8-27B on a LAN box wins for generic tasks on hardware that can hold it. Needle is for hardware that can't, like plain ARMv7, MIPS32 (the Ingenic chips in cheap IP cameras), RISC-V and watches. Also, we found cost to not really be the lever for on-device models, but availability and latency.

Strong point! Needle is a task-specific model and bullet 6 stressed that it is only trained to be good on a set of narrow tasks, but I guess it could be clearer?

thanks for this actually, so the demo is a preset, you can edit tools descriptions and add available tools the way you want, else Needle heavily guards against false negatives, users asked for this. For this, we will update presets on our end.

Ok we'd be adopting Little Language Model officially haha

Now you can try running the python package directly on your Mac to see proper speedup. WebAssembly has some limitations, running on pure hardware, especially ARM-based to see thousands (which 99% of consumer devices are ARM).

Please give it a try and let us know!

Haha, my rent is cheap lol

yes, that's what we had in mind while building

Ok, this is genuinely funny, we will fix these as we iterate, thanks lol.

Insane!

Your thinking is correct haha

The problem is the target device, an LLM can't run on an average TV well.

Thanks! Let us know how it goes :)

SO when confused, it gives you a low confidence score as a signal, in which case its recommended to finetune using th python package, train on your mac/PC

You explained it better, we've done a bad job at communicating its nice :(

Yep

Exactly the use cases we designed Needle for!

Thanks!

7 for now, English, German, Polish, French, Dutch, Latin, Italin.

A lot of Needle 1 users deployed on microcontrollers, we couldn't just throw them under the bus only t chase benchmarks, so its a price we are willing to pay :(

Haha, its not a generalist model, cheap devices cannot run LLM in production, so we brig on-device intelligence to them in the size they can. Needle is niche, not a Gemma or GPT replacement.

So TFLite is an inference engine while Needle is a model, which you can port on TFLite btw.

thanks, give the playground a go and let us know how to improve!

Yes, though for better results in production, after creating your tool json, use the provided data synthesis and fine-tuning pipeline. It tunes on on your mac.