Practically any criterion X you might use to restrict an ad's audience will have non-uniform correlations with group demographics. You can't wish away these correlations: they're true facts about the world. These non-uniform correlations allow activists to argue that allowing advertisers to target based on X is therefore prohibited discrimination. They can repeat this process for all X.
Now, you might argue that ad targeting is bad. That's a common position on HN. But it's not true: ad targeting is good for advertisers, since it enables more efficient advertising, and it's good for users, since it subjects them to less noise and exposes them to ads more likely to be relevant to them. Ad targeting also undergirds practically the entire tech economy. That probably means you, reader of this comment.
Where exactly should restrictions on ad targeting stop?
EDIT: For clarity:
Step 1: HUD files a complaint against Facebook for allowing ad targeting based on certain criteria.
Step 2: Facebook bans advertising targeting based on the criteria in the HUD complaint.
Step 3: Advertisers react by targeting based on demographic correlates of the categories from step #1.
Step 4: HUD notices that ads are still getting delivered more to one group than another.
GOTO STEP #1.
There's no clear point at which this process stops. Demographic correlates are numerous and strong. Are you going to ban all of them? That's tantamount to banning ad targeting generally!