The less the government messes with your freedom the better, eh? I suppose that if I didn't like the color, perhaps I'd repaint your house in retaliation. What a mess it'd all be.
My position is that visual pollution is not such a great concern that we should restrict how someone paints their building.
For example, if a world government said we don't want anyone wearing yellow hats. That's a problem because over the globe there are going to be some odd folks that care very much about their yellow hats (insert hypothetical religion / cause here). But if you get together with your neighbours and say let’s create a pact to legally require that all people remove their yellow hats from our gated community, well fine if you love yellow hats, you probably don't like these yellow hat haters anyway, who cares if they don't let you into their street.
That same could be done in local areas. But I think central business districts or areas that have a huge number of commuters there are just too many people involved. You now must have a lot more of a justification for the freedoms that are lost.
That said if all the building owners want to put that restriction in I'm all for it.
Besides, there is a perfectly libertarian or even ancap way of looking at the advertisement problem. Billboards affect the quality of a city, and most of the people it affects aren't getting paid proceeds. So they can gang up together to ask for a better deal. And if they can't then they are being restricted by some non-libertarian force.
You're correct about that, but not in the way you think. This site actually tends to lean more libertarian than most people in the real world.