In addition to that by far most things people would want to put in public spaces isn't explicitly designed to upset you like ads are. Why do we allow companies to plaster public spaces with veiled insults?
Usually buildings have private property owners. They agree to putting a bill board on their wall or they don’t. Graffiti sprayers usually don’t ask for permission — and that’s where the difference comes from.
In Europe you don’t usually have huge bill board on buildings. Rather you have lots of advertising columns on the side walks (here in Berlin we call them „Litfaßsäule“ named after the local inventor Paul Litfaß in 1854). You could argue they being a nuisance for sure but before the internet and even before radio and tv it was an important place of public communication. Actually they were an improvement because prior to advertising columns advertisers were wildly plastering anything with posters.
And yet if I agree to have someone stand on my property shouting insults at passers-by I'll get a visit from the police soon enough. This issue isn't as black and white as you make it out to be. Just because the owner of the building the billboard is attached to agrees to have it there doesn't mean no one else is affected.
There's already limits to what you can put on a billboard. Banning billboards isn't some radical new category of thing. it's simply moving that threshold down to "you can put nothing on billboards"
If it came to a honest vote on whether people would like billboards or no billboards I think the result is obvious.
A vote of people liking billboards is completely independent of the “oh, the powerful corporations use them” pearl clutching. I would hate billboards if they were just dominated by individuals using them to promote religious and political ideologies.
The only reason their usage is dominated by businesses is because businesses generate returns off of advertising. They don’t have special privilege and they certainly aren’t out of reach of individuals, clubs, non-profits, etc.