Hacker News Reader

Comment by snrji | Hacker News Reader

By no means will I be ever able to grasp Friston's theory, but the free energy minimisation vaguely reminds me of Curiosity-driven reinforcement learning. Can anyone with more understanding than me confirm or deny this apparent resemblance?

View on HN

There are similarities. The difference in two approaches are: - FEP is Bayesian in nature, while there's usually no notion of uncertainties in curiosity driven RL - In FEP, there's no explicit weighting of explore/exploit tradeoff. It automatically emerges from equations - FEP, since it's Bayesian, allows for more complex reasoning (like counterfactuals) - Curiosity driven RL is scalable while FEP is not feasible for anything other than simple models

Many thanks for your answer.

Excuse me, another followup question (can't edit on mobile): can you ELI5 how do exploitation and exploration "emerge" naturally instead of the tradeoff being explicitly coded as in RL?

In forumations of FEP, there are two terms: cost and ambiguity. Minimisation of this combined term happens in a Bayesian optimal way. So you don't have to explicitly code weights for exploration and exploitation.

Although what you do have to code is prior preferences, and since it is a distribution, you implicitly code the range of those preferences. But once you do that the FEP, algorithm figures out when to collect more data to build a better model and when to use the existing model to get near the prior preferences.

I see. Much more elegant than explicitly coding the trade-off, actually :)

As a general answer, the theory suggests that organisms maximize a quantity known as model evidence, which is just a way of saying 'how much evidence does some data provide for my model of the world?'

There are two complementary ways to maximize this - change your model or change your world.

If we now grant that actions also maximize model evidence, then actions can either be conducted to sample data that make the model a better fit of the data (exploration), or they can be conducted to sample observations that are consistent with the current model (exploitation).

And the optimization process itself would determine whether updating the model or changing the world is optimal, I guess. Thanks.

The equation for free-energy/ELBO has two terms, an energy and an entropy. You can rewrite it as "log-likelihood minus KL from prior". If you write your model in a certain way, you can then read it as, "Fit to the data, minus cost" (second formulation) or "Accuracy + exploitation + exploration" (first formulation).

There is a large overlap, for instance, the popular VIME exploration algorithm [1] uses part of the free energy objective function.

However, free energy isn't a theory of curiosity per se, its posed as description of self-organisation. It just so happens that you can express the free energy functional in terms of epistemic (curious) and instrumental (reward) components.

[1] https://arxiv.org/abs/1605.09674

I see, thanks. Is any AI research center/company trying to model this idea, instead of applying RL?

Excuse me, another followup question (can't edit on mobile): can you ELI5 how do exploitation and exploration "emerge" naturally instead of the tradeoff being explicitly coded as in RL?

That has been my go-to comparison for a while as well, and it sounds like what they refer to in this article when they talk about using active inference-based AI vs classical reinforcement learning.

Yes, although looking at what other people have commented, "our" intuition is quite basic. I will try to read more about FEP.

Reply on news.ycombinator.com