This is interesting :-)
Is there any reasonably accessible literature on this topic, in particular the latter idea of not being able to estimate an agent’s utility function?
Is there any reasonably accessible literature on this topic, in particular the latter idea of not being able to estimate an agent’s utility function?
1. Von Neumann and Morgenstern's book (particular the sections where they develop the notion of utility) is pretty readable - https://www.goodreads.com/book/show/483055.Theory_of_Games_a...
2. The Economic Naturalist (https://www.goodreads.com/book/show/629238.The_Economic_Natu...). It is a great compilation of economic edge cases which show that the notion of utility is not so easy to capture using the standard "narrowly self-interested economic agents" world view. It requires a lot of subjective interpretation.
3. The Honest Truth about Dishonesty (https://en.wikipedia.org/wiki/The_Honest_Truth_about_Dishone...). Shows how contextually dependent utility (if it is at all quantifiable) can be.
4. The body of work around the Iterated Prisoner's Dilemma: (https://en.wikipedia.org/wiki/Prisoner%27s_dilemma#The_itera...). In particular we can see how the dynamics of simulations using rational agents diverge from the dynamics of experiments involving actual humans. This suggests that the utility assumptions used in economic analyses do not reflect whatever notion of utility that could actually apply to human behavior.