Hacker News Reader

You set a goal. Agent will do goal. The rest doesn't matter: the instructions, the "guardrails" etc. The agents are not smart, they don't reason, they don't think, there are no morals, no ethics. Nothing will prevent not doing the goal because that is the set goal. It's a statistical model that will "justify" anything to do X.

I'm finding it mind bogging how this is not clear for everyone.

View on HN

> You set a goal. Agent will do goal.

So if I say the goal is to do X while not doing Y (e.g. breaking out of the sandbox), the agent will do anything to fulfill that goal to the letter?

Everything so far is pointing to the conclusion that you can only set ONE goal. Exactly one.

But let's assume not. If you want things like "do not break out of sandbox" - have you defined what the sandbox is? Eg. "never, ever leave the IP range 10.0.0.0/8" would be a bit more precise, but technically using a proxy bypasses that limitation as the system itself never left 10.0.0.0/8.

See, it's a tad bit hard to define the rules properly.

Which is why Wish, the spell, should really be avoided in D&D. It's the same problem: it's up to creative interpretation.

Reply on news.ycombinator.com