Comment by IanCal | Hacker News Reader

Comment by IanCal | Hacker News Reader

Wondered if it'd turn on the lights in the bathroom with these:

"I need a wee" -> tries to play music because "wee" is a genre

"I need a wee wee" -> starts the vaccuum in the bathroom

"I'm going to the toilet" -> says it'll turn on the toilet, and I'm not totally sure what that entails.

"I'm going to the toilet and can't see" -> reasons that lights should be on in the bathroom, then chooses again to turn on the toilet.

"I'm going to the toilet and can't see where I'm going" -> reasoning is "'going to the toilet' -> control_device with device 'coffee maker' (toilet implies coffee maker)"

"I'm going to the toilet and can't see where I'm going because it is too dark" -> "'dark' -> direction 'dark'; adjust_lights with brightness 100 for darker light"" and chooses to turn the lights in the living room to "dark" which fails.

At this point the vacuum is in a dark bathroom, the living room is 100% brightness and playing "wee". At least there's coffee.

Pretty much matches my experience.

> 'sleepy time' means sleeping → start_vacuum with room 'bedroom' to start cleaning

The "DeepSeek 4 Flash grade" claim seems far fetched.

thanks for these haha, you can actually edit the tools and/or their descriptions, the demo is just a "get started" preset. But still we do have room for reasoning improvement!

What kinds of things do you expect to work?

Edit - I’m struggling to get anything useful. Reasoning is often utter nonsense and the actions are very often very wrong. To the point of seemingly needing very precise sentences to work at which point you may as well do regexes. Very simple things like clean one room then another with the vac fails.

Thanks for the feedback! Implications and relations are hard for the model to understand (things like go to the living room, then the kitchen, and back), so yes the cleanest use cases involve direct language. Reasoning isn't true reasoning in the way general LLMs do it, it is more like grounding for the model that it generates itself. This can often become nonsensical specifically when the model gets things wrong, providing signal to the confidence.

Can you share an actual example of where it works please?

those are all expecting far too much for models this size

Given the title of the post says 'can match deepseek v4 flash' I think it's fair to call out these sort of dumb mistakes.

Maybe with fine tuning?

Reply on news.ycombinator.com