I am all for being critical of class-based inequalities but having resources does not exempt one from adversity. Very few people would consider someone who spent their whole childhood in debilitating sickness, someone who lost their wife and mother 2 days apart, rode western with cowboys through a winter in North Dakota as someone LARPing in controlled environments. Not all of this is romantic ideology. One can be born with a silver spoon and still experience the realities of life, the trauma of losing loved ones, prolonged sickness, etc. Saying that he "never faced any actual adversity" is black and white while being arguably wrong.
The Edmund Morris trilogy gives a good overview of his life for those interested.