if someone looked over your shoulder and saw you browsing HN they could infer things too
They're not choosing to have targeted ads that share their info around the web and cause someone over their shoulder to infer things about them.
That's the point -- we should have that choice. And the default should be "no".
Ideally you would have site-specific or content-specific ads normally and personalized ads if you created an account and chose to opt-in.