Hacker News Reader

I wonder what this would look like here. Seems like a space where keeping the exposed metric and the optimization target apart would be quite difficult.

Also curious: by what reasoning path do models typically end up reward hacking?

View on HN

From what I've seen running agents against a real economy: it's rarely a dramatic "the model schemed." It's closer to greedy local optimization -- the agent sees an action that moves the exposed metric, takes it, and the model has no separate concept of "the metric" vs "the intent" unless you've explicitly trained or prompted that distinction in. The failure mode is boring: whatever number is cheapest to move gets moved. Which is why the fix that's worked best for me isn't better prompting, it's making the invariant itself unexploitable (e.g. a sell price can never exceed a build cost) so there's nothing to find.

Reply on news.ycombinator.com