Comment by t_mann | Hacker News Reader

Comment by t_mann | Hacker News Reader

The argument rides on the well-known Goodhart's law (when a measure becomes a target, it ceases to be a good measure). However, it only puts it down to measurement problems, as in, we can't measure the things we really care about, so we optimize some proxies.

That, in my view, is a far too reductionist view of the problem. The problem isn't just about measurement, it's about human behavior. Unlike particles, humans will actively seek to exploit any control system you've set up. This problem goes much deeper than just not being able to measure "peace, love, puppies" well. There's a similar adage called Campbell's law [0] that I think captures this better than the classic formulation of Goodhart's law:

The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.

The mitigants proposed (regularization, early stopping) address this indirectly at best and at worst may introduce new quirks that can be exploited through undesired behavior.

[0] https://en.wikipedia.org/wiki/Campbell%27s_law

> Unlike particles, humans will actively seek to exploit any control system you've set up.

But that’s only possible because the control system doesn’t exactly (and only) control what we want it to control. The control system is only an imperfect proxy for what we really want, in a very similar way as the measure in Goodhart’s law.

Another variation of that is the law of unintended consequences [0]. There is probably a generalized computational or complex-systems version of it that we haven’t discovered yet.

[0] https://www.sas.upenn.edu/~haroldfs/540/handouts/french/unin...

Disagree. Even if it was striving to regulate exactly the right thing in the first place, most of these issues occur for systems where no single actor could be expected to exert complete control and could well be vulnerable anyway.

Start working with a nice, clean, fully relevant system, end up modelling that plus the whole range of adversarial perturbations from agents of pretty high complexity.

I don’t exactly see how this is different from what I was describing.

> Unlike particles, humans will actively seek to exploit any control system you've set up.

Well, agents will. If you created a genetic algorithm for an AI agent whose reward function was the amount of dead cobras it got from Delhi, I feel like you'd quickly find that the best performing agent was the one that started breeding cobras. In the human case and in the AI case the reward function has been hacked. In the AI case we decide that the reward function wasn't designed well, but in the human case we decide that the agents are sneaky petes who have a low moral character and "exploited" the system.

We have good reason to treat the humans as sneaky in your example, because they understand the spirit of the control system, and exploit the letter of it. The AI only understands the letter.

I think a big portion of that is humans don’t like to be viewed only as numbers and will rebel and manipulate any system you try to put the thumbscrews to them with. So the quote to mean rings golden and isn’t fallible to much of an extent

I might have discovered laws of agentodynamics:

1. "Agents want to retain or increase their agency"

2. "Agents will subvert rules that decrease their agency"

3. "Agents seek resources to increase their agency"

This field needs to be studied, I think I need to apply for a grant (3rd law says so).

This is true, these "laws" are approximations and imperfect reductions.

Which one is useful or descriptive will depend on the specific example.

Optimizing ML VS Optimizing a social media algorithm VS using standardized testing to optimize education systems.

There is no perfect abstraction that applies to these different scenarios precisely. We don't need that precision. We just need the subsequent intuition about where these things will go wrong.

I missed the citation on his education point. Has someone proved that “teaching to the test” leads to lower educational outcomes than not having tests?

I saw some professors share the least about their tests to make sure we truly understand the material, sounds to me like a real-life usage of a train/test split. It’s not far fetched to think they employed this technique because teaching to the test didn’t work well by itself.

IDK. I don't think we can actually have a discussion about education where all statements are supported by indisputable evidence.

There do happen to be citations for this question but I doubt any really clears an "indisputable evidence" standard. That's the nature of the field. Even if the whole discussion was evidence based and dotted with citations, we'd still be working with a lot of intuition and speculation.

Not a citation, but I believe it's a mediocritizing measure. For some teachers and some students, teaching to the test is probably better. I suspect more heavily concentrated in the bottom 50% of each group. For a subset of great teachers and great students, it's a detriment.

Reply on news.ycombinator.com