Link to the actual paper. https://pathak22.github.io/large-scale-curiosity/

I was curious how they define or reward curiosity, it says it right here:

Reinforcement learning algorithms rely on carefully engineering environment rewards that are extrinsic to the agent. However, annotating each environment with hand-designed, dense rewards is not scalable, motivating the need for developing reward functions that are intrinsic to the agent. Curiosity is a type of intrinsic reward function which uses prediction error as reward signal.

So, the prediction error is the reward, nice.