I can think of a few more downsides, especially if one considers more factors than personal material gain and comfort.