Adding to the analogy: The current owners of the home regularly remove swastikas spraypainted on their house by vandals, and some people call this a violation of free speech.