Comment by soycaporal | Hacker News Reader

Comment by soycaporal | Hacker News Reader

Hobby project, I wanted to "ship a useful model in a web browser". so I distilled a small sentence encoder from MiniLM with ternary quantization-aware training. Also wrote the inference engine from scratch and shipped in Rust → WASM SIMD.

It's an embeddings model, not an LLM: text goes in, a 384-dim vector comes out, and cosine similarity between two vectors tells you how related the texts are — regardless of shared words ("reset my password" ↔ "I forgot my password" → 0.88). Used for semantic search, FAQ/intent matching, and clustering. Running it on-device means search-as-you-type semantic search is performant with no API dependencies.

Demo (2k React docs, fully on-device): https://ternlight-demo.vercel.app

Two tiers on npm: - @ternlight/base (7 MB, ~5 ms/embed, more capable embedings) - @ternlight/mini (5 MB wire, ~2.5 ms/embed).

Bundled for Node and browsers.

Repo - see technical details (MIT, training pipeline included): https://github.com/soycaporal/ternlight

Curious if this is something useful, what are the use cases for on-device embeddings.

Awesome. We have a dictionary of words to OpenStreetMap tags here : https://codeberg.org/cartes/web/src/branch/master/components...

Do you think your work could help us let users type "pancake" and get "crêpe" without writing an explicit "pancake = crêpe" dictionary entry ?

In practice : if I understand well, your lib would first need to download 5 Mb, once and for all, and would then be used as we use Fuse.js right now ?

How well does it handle languages other than English ?

Could it be "trained" on the OpenStreetMap tag wiki ?

Thanks a lot for your work.

The OSM tag wiki would probably not make for good training data because it only has a single short description for each tag (and even on the French wiki many descriptions seem to be in English?) whereas you want to map multiple descriptions to the same tag.

Ideally you would have real query data (e.g. from cartes.app telemetry), then you could get a LLM to write a bespoke Overpass query for each one and use that as the ground truth. Alternatively, start from the list of OSM tag values used in the wild and ask an LLM to list possible reasons to visit that POI.

You could then use that data to finetune an embedding model for your use case. But, you know, somewhere in that model there's going to be a token vocabulary that the model knows about and at the other end you get a similarity score for each tag value. If you don't need to support complex queries where interactions between words matter ("any restaurant that is NOT Korean"), you could get away with a simple list of words and tags that they match to. Which is right where you started, except it could be more exhaustive. Why limit yourself to two Korean dishes when you can have a LLM list many more for you?

Thanks a lot !

The corpus is mainly trained in english, unfortunately no other languages have been included in the distillation training. Yes it would work like fuse.js, but unlocks semantic search.

Source code has the entire embedding distillation pipeline includes dataset preparation. You could run the same distillation training (but not sure if teacher model used a multi-language dataset).

Open up an issue on the github repo and I can reply with details

Thanks! I strongly suggest copy-pasting that explanation to your web page, that's a nice summary.

Nice, I'm really interested in using this for simple semantic search in a native desktop application.

Any comparisons with other tiny embedding models? Did you start from MiniLM-L6 because it's an especially good model in its class? It's hard to figure this out since all you provide is "Retrieval (SciFact NDCG@10)".

But the claimed performance seems way off, I get only 35 emb/sec in firefox on a i5-4570 rather than 400/sec. Is there an issue with falling back to a non-SIMD path? I'll try a native Rust binary next.

Same! I’m trying to find small models that can embed effectively to enable BM25/hybrid search over a large number of documents for a personal information repository. Ideally, it should run on consumer hardware.

bge-small-en-v1.5 is one that is comparable and what we’re working with for now.

the base model I clocked it at 5 ms per embedded on my mac studio. There is a mini variant (the demo version) that is sub - 2 ms. It could be a SIMD issue.. I'll look into this for better runtime support (also fee free to file an issue)

0.84 Spearman fidelity to the MiniLM teacher at ternary precision is a striking result. How much of that is the quantization-aware training doing the work, versus what a post-training ternary quant of the same encoder would give you?

It's entirely the QAT. The whole distillation process is quantization-aware from the start, so the ternary weights are learned rather than fitted after the fact.

The only post-training quantization I applied was int4 on the embedding layer, and I ran a small ablation there to find the sweet spot between size and quality.

Huge kudos for sharing everything including your training code. Awesome project!

Awesome! Besides size, how does this compare to gte-small?

gte-small outscores all-MiniLM-L6 on MTEB (~61 vs ~56 avg per the GTE paper). MiniLM is ternlight's teacher (ternlight holds 0.84 Spearman fidelity to teacher). I haven't run a head-to-head yet; STS-B/MTEB numbers are on the roadmap. Also on the roadmap is to distill gte-small as teacher.

Thank you for this tool!

We've just used it to embed the entire django doc + our private knowledge base, allowing us to search in the 2 sources instantly!

super cool use case! Hopefully it can provide quality embeddings + retrieval. Would love to learn to results/issues or feedback. Please feel free to file for issues on github

What is the process for adding different text? What are the limitations on that process? The demo is very cool by the way.

Reply on news.ycombinator.com