Comment by gs17 | Hacker News Reader

Comment by gs17 | Hacker News Reader

"turn all the lights on/off" and "it's too dark in the bathroom" worked for me, but anything less direct didn't. "it's too cold" actually made it turn the thermostat down ("it's cold" made it... turn the lights down?)! Although the confidence on the bad responses was pretty low, so it might be worth adding a threshold to the demo.

Or maybe it just has a weird thermostat down bias? "make it hot" also had it turn it down (specifically it went from 20->18, or at least tried to, the UI still showed 20), with high confidence. Also might have a bit of a Celsius vs Fahrenheit confusion. Neat concept, but I might not want to let it control the oven at the moment.

The laptop demo worked better until I tried to open the mail app. "Check mail" kept opening the browser with an error, and "check email" makes a note with the text "email", "open email" goes to "https://api.email.com/v1/email" in my real browser, but "open mail" does work.

And I presume the "reasoning" isn't very trustworthy? In the car I got "'turn it up' means lower volume -> set_volume with lower value." For the house, reasoning would correctly say that I wanted the alarm off, but it didn't actually do it.

Hey, thanks a lot for this feedback, very useful and actionable for us! Quite a few of these came down to our tool definitions in the playground as well as out triggers. We updated them just now and these should be more reliable. Really this goes to show that needle shines through after putting in the work to make the tool list around it good for your use case. As for the reasoning, yes its main function is really to provide more words/keywords that the model can latch onto when generating the tool call response, since this is a SAN model it needs more grounding in existing context.

I can’t help but wonder how well more traditional approaches would do with this. Something like a map of statements to actions, with fuzzy search - then remove what used to be the labour intensive part of this by handing it to a decent llm to generate the sentences.

That's a really good point and I think it's not yet clear how well, say, 8-30MB worth of regexs with accompanying algorithmic structure would do on these tasks. I would imagine they do quite well on a well defined task, but it would be much harder to then adapt this set to a new domain. A big part of Needle's promise is how easy it is to finetune. Ultimately I think the two approaches can be more complimentary to each other, rather than choosing only one (see triggers!).

I think a good "traditional" approach would look like a BM25 algorithm over an index of trigger phrases for each category, sitting behind a majority-vote classifier. The "fine tuning" would be done by reindexing the data, generating different/new phrases, and tuning the classification threshold.

I think we might look into creating a baseline like this for our future models

Reply on news.ycombinator.com