Reward Is Enough (2021) | Hacker News Reader

Reward Is Enough (2021)

sciencedirect.com

21 pointsoptimalsolver15 comments

"Reward Is Enough" is an interesting paper, less for its content, more for its place in the zeitgeist or something. It has very little in the way of novel "theorem proving" but it has sparked quite a lot of discussion. It's a paper which manages to be really great in some sort of subtle way, just because it inspires so much conversation.

I'll plug my own response, "Can reinforcement learning learn itself? A reply to 'Reward is enough'", in which I ask: ok, if creating good enough RL agents can lead to AGI, and if AGI can do anything humans can, and if humans can write good RL agents, doesn't that seem to imply it should be possible to use an RL environment to incentivize RL agents to write good RL agents? https://philpapers.org/archive/ALECRL.pdf

>if creating good enough RL agents can lead to AGI, and if AGI can do anything humans can, and if humans can write good RL agents, doesn't that seem to imply it should be possible to use an RL environment to incentivize RL agents to write good RL agents?

Juergen Schmidhuber and Eliezer Yudkowsky are suddenly knocking on your door.

Of course it can, just like a Turing machine can simulate itself.

I for one, would greatly forgo reward if it were to avoid suffering. I don't think intelligence is simplistic enough to be driven by just a scalar field. I think like physics, we'll likely need two scalars, or a vector, or a spinor, etc.

If "-1" is suffering, and "+1" is reward, then they cancel out. In my own experience, suffering and reward do not cancel out. They have to be on different dimensions. What are your thoughts?

In studies it seems to take about 5 positive experiences to undo the mental impact of 1 negative experience. [1]

So yes - generally these are not considered equal. From almost every perspective, this also makes sense - trading 1 for 1 doesn't lead to you improving your situation.

[1] https://www.seekingbalance.com.au/wp-content/uploads/2016/06...

The claim here is even stronger, and somewhat different: that suffering and reward are fundamentally different kind of things, and cannot simply undo each other, not at any exchange rate. The difference is qualitative, not quantitative. For example, if you lose your eyesight, it's such a big loss, that most likely nothing can balance it out ever. It has no "five times as good" equivalent.

This is exactly what I meant. Thank you for providing a much more accurate and elegant description.

High cost of suffering (e.g. being ran over by a car when street crossed on red light) and relatively small reward for corresponding success (meh, just crossed the street in the example) might explain it.

This just points to a different way of defining the reward function, no? You would weigh suffering with a high negative weight.

In the simpler case you are optimizing on a single scale between maximum suffering and maximum happiness.

In the proposed, more complex case you have two independent scales, the suffering scale (on which you want to primarily minimize), and the happiness scale (on which you want to secondarily maximize).

"Negative utilitarianism is a form of negative consequentialism that can be described as the view that people should minimize the total amount of aggregate suffering, or that they should minimize suffering and then, secondarily, maximize the total amount of happiness. It can be considered as a version of utilitarianism that gives greater priority to reducing suffering (negative utility or 'disutility') than to increasing pleasure (positive utility). This differs from classical utilitarianism, which does not claim that reducing suffering is intrinsically more important than increasing happiness."

https://en.wikipedia.org/wiki/Negative_utilitarianism

In psychology/behavioral economics this is known as loss aversion. Most people would rather avoid losing $5 than not earning $5. But given the rather broad view of reinforcement learning this paper adopts, I would guess that the authors would not view this difference in sign as important. (And indeed, in the more ML/AI view of RL, viewing reward as a negative loss or cost is quite standard and for the most part the sign is seen as an arbitrary convention).

In RL, reward does not mean positive only. You could train an agent on purely negative rewards and it would optimize for the policy that is least negative.

Also, +1 and -1 do cancel each other out by definition. If your model has suffering greater than succeeding, then you would assign suffering a greater penalty, like -10 vs +1.

Science Direct would think that given that all their reviewers and authors are paid zero dollars but Science Direct turns a profit.

If you define reward as dollars, then the reviewers and authors receive no reward.

Reply on news.ycombinator.com