My understanding is that it was classism. That is, only the "right" people could dedicate themselves to becoming a world-class athlete without any financial support, and they didn't want to complete with the "wrong" people. I think you can see similar in how "prize fight" boxing was initially looked down upon as a corruption of gentlemanly dueling.

(Can't find sources for this right now, sorry. And I'm not even positive I am remembering it correctly. So perhaps take my response with a grain of salt.)