If you're an opponent of increased housing supply in California, you can roll out that talking point whenever threatened. It's tested, emotionally resonant, and most people don't have the time to be explained why it's wrong.
Supporters of increased housing in CA need to come up with their own powerful rhetoric, rather than whining about the fact that their opponents' rhetoric works.
(Whining further that bad rhetoric shouldn't be allowed is even worse--we live in a democracy; you have to influence people; rhetoric matters.)