My take is more that the framing of maximizing an externally provided reward describing a single task only helps an agent who can rely on the rules of that task remaining fixed. Humans don't just try to do one task well; we switch between tasks. We don't have episodes that reset really; we have one long episode, and our actions decades ago can have bearing on our well-being today. What is a good state or a good policy varies substantially over time. The concept of the value of a state or (state, action) is ambiguous, because who knows what you'll want tomorrow? If the definition of value shifts, but the rules of the world don't, then a modeling becomes more "valuable".