It's really not that hard to see how many would lose a whole lot if this feature was available.
If side-loading was allowed and tons of users side-loaded pirated games I'm sure these users would gain, but would the developers? Wouldn't they call on Apple to stop this? I mean the headlines just write themselves: "Apple is complicit and profiting from piracy by selling phones on which it's trivial to install pirated games".
If people started installing random software they heard of on social media, how would the image of the iPhone be changed by the inevitable malware that people would install? "This new iOS virus steals your data and private photos".
What about all the spyware apps that would get side-loaded onto the phones of unsuspecting spouses? "Woman murdered by psycho ex who tracked her every move through her phone".
It takes a little bit more than "it's silly we would all only gain" to make this argument.