There's a ton of freedom already taken from you for the benefit of society, like driving on one side of the road or stopping at intersections. I don't think you mind those?
Each case has it's own cost benefit. I think the road rules have benefits that outweigh the costs. I don't think the government should be able to dictate how people dress their building. Be that painting them bright green or putting up advertising. Just as they shouldn't tell me how to dress.
Governments do have rules about the look of buildings. There are zoning regulations about commercial vs residential properties. Many places around the world will require that your building conforms to the character of the local environment.
A precedent doesn't prove it's an optimal solution.