If they opened a pop-up restaurant or nightclub, I'd be pissed too; this wasn't a contract to sublet a space for commercial use after all. You can get libertarian about sex work, while ignoring the abuse of an agreement between to parties.
While the brothel example might not be distinguishable from a couple on vacation having a party, except for the money-changing-hands angle.
Lets admit, its the moral angle at issue. The rest is after-the-fact arguments pasted on to make it seem like an objective thing.
As to wear and tear... it's a brothel... and not in a movie; you might be surprised at the turnover.
Edit: I realize that "have no patience for" sounds pretty snotty... I've been a little sleep deprived lately and it isn't bringing out the best in me, sorry.
Edit 2: I also realize that the issue of the brothel, totally aside from the homeowner's feelings, might represent a property value issue, in the same way as a previous murder. It doesn't have to sensible for it to hit him in the pocket; obviously sex work doesn't leave some kind of "spiritual residue" anymore than violent death, but people are often irrational.