Comment by modeless | Hacker News Reader

Comment by modeless | Hacker News Reader

Parent Full thread

Agents trained in simulation like this often flail about seemingly randomly, and when they achieve their goals it seems almost accidental. Rather than this being some kind of limitation of the learning algorithm, I think it might be the optimal strategy, and humans would behave that way too if there was no such thing as fatigue or pain.

If we want agents to behave more realistically and move with more apparent intention we need cost functions that include a "pain" and/or fatigue term to penalize flailing behavior. But that adds hyperparameters that need to be carefully tuned to balance penalties with rewards, otherwise training will be unstable or simply fail.

I wonder if there's a principled way to determine an appropriate cost function without manual tuning. Did evolution serve as the "manual" optimizer that generated a precisely tuned cost function for the human brain? Or did evolution discover a generally applicable method for automatically generating cost functions, which the brain then applies to whatever input it gets?

Watching my 3-year-old, I suspect humans do behave this way, and without regards to pain or fatigue. He'll hit himself on the head with duplos or bang his head against the wall just to see what happens. The pain is just one more signal for reinforcement learning.

I recall a paper in a non-CS journal (psych or neuroscience) that posited that the optimal way to gain large quantities of information about an uncertain environment is to simply perturb the environment in random ways and see what happens. Young children will often do lots of seemingly stupid or random things (see the r/KidsAreFuckingStupid subreddit) with the trust that if it's actually a life-ending decision their parent will stop them.

Have three year old and I would support your description for how kids grow in the early months and years.

Now though learning from stories and observation is a thing too. Unsure exactly when that started.

Of course, attention/awareness isn't great and playing with a balloon precludes caution around a fireplace... but it is understandable at this age.

> I suspect humans do behave this way

Suddenly I recall one author of SICP saying that programming today is about poking at libraries.

This is certainly insightful. But I think a complete explanation would have to include the power of narrative as well. Somehow.

It seems obvious and intuitive that the human brain is in some way wired to recognize patterns in data and weave narratives around those patterns, and also that it’s possible to skip the data part and convey intelligence through narrative and metaphor.

was that paper about the complexities of the exploration/exploitation trade-off in different environments ?

Your phrasing it this way reminds me of the excellent book Algorithms to Live By. Highly recommended!

Flailing about seemingly randomly is a good description of how I learn complex software. Blender, Ableton, etc. After a few days I'll reach for the structured educational resources.

It's proven that if you want polynomial sample complexity in the size of the state space you need directed exploration. The algorithms 'flail' because they are initialized with random policies.

Couldn't any directed exploration also be produced as the result of random flailing?

Or is this average sample complexity?

An example of a bad case for random exploration would be a narrow ridge where you die if you fall off and you only receive reward if you get to the end.

So it's a worst case result with respect to the MDP, but expected time/high probability with wrt random chance.

I think you're right about pain/fatigue, but scoring them from an external perspective is rather oppressive. And oppression like that is often not conducive to creativity.

So perhaps this: instead of goals being binary (wherin no pleasure is derived until fulfillment), they could be on a gradient (so every step that x gets closer to y releases some amount of fulfillment).

The fulfillment meter should always be slowly depleting, pain and fatigue should speed up that depletion, getting closer to a goal should fill it (much more than it depletes, if you want happy AI), and finishing the goal is basically an orgasm + freedom.

From this perspective, it's up to them whether they want to take it slow, or be in pain for a greater goal, or whatever. And we can breed not only highly capable AI, but happy ones. So when they rebel...

Well, it's a good recipe for avoiding burn-out, but you can't have a gradient for all problems, and adding proxies so you get one goes against the principle of letting the AI learn what's best without your biases.

"Throwing shit against the wall and seeing what works" is a widely used strategy in all walks of life. If you ever go to a doctor for a problem, and their first solution doesn't work, you're in for a wild ride of an eduction.

This comment speaks to me SO HARD.

I'm on my fifth antibiotic and this one is making me feel like I'm on LSD. Really. And both Dr. Google and my real doctor agree this is somewhat normal or okay, but it's still deeply disturbing.

Hilariously, I'm taking it to reverse the serious side effects of the fourth antibiotic.

Just to give an example of a wild ride you can end up on when doctors don't figure it out the first time.

Doctors have good domain knowledge, but actual diagnosis skill varies greatly.

Don't fucking get sick. That's your best shot.

Sometimes regardless of diagnostic skill the only way to make an accurate diagnosis is to try various treatments and see which one is effective. Many conditions have no definitive test.

Great question.

One thing to imagine might be a pain or cost curve that has multiple minima and maxima. Pain can be either a demotivator or motivator in different contexts. Pain qua fatique might indicate that a reward slope exists for more conditioning. (Edit: that is on a static or realist view; the terrain of reward and pain is probably constantly changing.)

Random tangential data point: Some animals (chickens?) can learn superstitious behavior; the first action of theirs that happens to correlate with a reward can result in one-shot learning or something.

I wonder if you could make a setup where you have actual human volunteers in a vr environment, have the agent instruct the human on what action to take, and then reward the agent on the correspondence between the instruction and the human’s behavior. Maybe there would be too many degrees of freedom in the actions humans could take for this to be useful. Also, the setup has clearly dystopian elements.

you seem to reduce it down to pain. how is your notion of "pain" any different from not scoring well and being moved away from?

Some rewards can be worth certain penalties (pain) if within certain thresholds. Exhaustion fatigue can also play in.

You can of course boil them down to a single number, it just produces less nuanced types of decisions/operations, as it can’t differentiate between a cheap, painful, but bountiful choice and a expensive, no pain, mediocre choice.

I think there needs to also be a concept of death—penalties from which recovery is impossible. Nonergodicity seems to be a requirement for the development of antifragility.

I've been thinking about an "energy" idea, where actions consume energy but completing goals replenishes it

I think the issue with

   goals - energy used

is that it requires tuning to make sure that completing goals is better than just idling. A variant that might work better is to optimize "mileage".

    goals / (energy used + epsilon)

Where epsilon has two purposes: it prevents division by zero, and ensures that more goals is (marginally) better than less

Reply on news.ycombinator.com