Hacker News Reader

Comment by r-johnv | Hacker News Reader

Thank you! This has been an exciting project to work on!

For the industrial robot (in the video on the main readme page) I intentionally gave Claude no context beforehand. All of the inferences that you see there are based on information that it got through the MCP tools that let it analyze the ROS topics and services on the robot.

In fact, I had a starting prompt to ignore all context from previous conversations because this looked to be like an example of emergent behavior and I wanted to confirm that it was not picking things from my earlier conversations!

View on HN

If you don't start from a clean-slate, does the behavior change drastically? Does the interaction between the LLM layer and the MCP/ROS become more efficient if it already has some context? Would that be something you'd want to toggle or do previous conversation contexts cause issues with new command implementations?

Not starting from clean slate improves the interaction significantly. I started from a clean slate on the industrial robot video in order to highlight how much is possible even when starting from one.

That makes this all the more impressive!! What happens when you get an incorrect interpretation though? That is now in the "previous context bucket". Assuming the user addresses the issue through the LLM layer by talking to through it, do you think that the subsequent interactions could compound the error?

I sometimes face issues with LLMs running out of tokens or only using partial contacts from previous conversations - thereby repeating or compounding on previous incorrect responses.

Any thoughts on how to tackle that? Or is that too abstract a problem/beyond the scope to address at the moment?

(Mentioning beforehand that we're still very early when it comes to the exact behavior of each language model)

So far, I've observed that for Claude and Gemini, which are what we've been testing most with, the Language model has been pretty good at recognizing its a faulty initial interpretation when it queries more information from the system.

Running out of tokens is a more significant issue. We saw it a lot when we queried imaged topics, which led us to try writing better image interpreters within the MCP server itself (credit to my collaborators at the Hanyang university in Korea) to defend the context window. Free tiers of the language models also run out of tokens quite quickly.

PS - Thank you for the questions, I'm enjoying talking about this here on HN with people who look at it critically and challenge us!

I'm actually reviewing a PR right now that enables the MCP server to have a library of common robots with a specification file for each robot containing important context to the language model.

The demo video with the Unitree Go (robot dog) uses this approach to give the LLM additional context of the custom poses available to it.

Reply on news.ycombinator.com