Thats actually why I argue objectivism as being aligning better with what's possible within the field of ethics. The whole idea is you can only morally do what you want, without intentially removing others freedom, unless they pose a direct threat to your freedom. As a rule of thumb, this actually works exceptionally well.
Murder for example is wrong (morally), by that code of conduct. Traffic laws would be moral because it keeps others from removing your freedom by either not letting you access the road or accidentally killing you by being reckless. It also handles the Hitler argument, whereas some utilitarians my argue what Hitler did was moral.
The whole point of this philosophy is to maximize personal happiness, while minimizing conflict. Data has little to do with it, because respecting one another is the whole point. When you start discussing ethics in terms of "sociatial good" it starts requiring one to know the future. For this reason, I'd argue objectivism, when applied, provides the more useful form of ethics, where one only has to worry about their own happiness, while not intruding (too much), on others.
Another problem is whether it's OK to require a thirsty person in a desert to give up all their assets in exchange for water, and whether it's OK for the thirsty person to knock you out and take the water in response; most people would side with the thirsty guy for what I think are very good reasons in terms of the practical implications while objectivists would not.
To your other problem, I fully understand the practical implications. Knocking the guy out and stealing his water is pretty unethical. However, coming to equitable terms is fairly easy. More humans can think one step ahead and just gouging the guy for all his posetions means he won't help you in the future. Long term, that is bad for you, thus you'll likely not charge too outrageous
For instance, why is making noises infringing on your freedom from noise, but the requirement of silence is not an infringement on your freedom to make noises? Freedom from noise and freedom to make noises are both "freedoms" and one comes at the expense of the other. What is the way to weigh their relative importance which is universally applicable to other cases and not rooted in culture or personal preference?