I'm not an expert in the field, but my understanding is that you can be (partially?) fingerprinted based on the ads you (fail to) download. I also believe such partially discriminating information can be combined with other patterns (time at which you connect, sites visited, etc) to compromise your privacy.
I have no idea if this is problematic in practice, though.