Sure, there's history there. But I don't see it as "damage". He is clearly quite happy when he is with his people.
His other great delight is figuring out how to open his palace in the morning to get out early. Every time he manages it he struts around for quite a while looking like he just won the Superbowl.