Data has a way of being used in ways other than the one for which it was ostensibly originally collected. Especially once you let data scientists get in there and start having fun with it. (Source: I should know, I am one.) Or maybe you've got a unscrupulous employee who uses their access to the data to dox people. (Source: I used to work at a company where that happened.) And since, in a country like the USA, they are not subject to any particularly effective data protection laws, they're also really easy to sell to just whoever, or maybe liquidate at a bankruptcy auction, or whatever. The buyer may or may not intend to use it for better or worse purposes than the original collector. There's no real way of knowing.
There's also the security question. Data breaches are real and happen all the time. I think that the crackers' perspective on this subject may be, if I may misappropriate the famous IRA statement, "remember we only have to be lucky once. You will have to be lucky always."
In short, the mere existence of these pools of data is a threat, not only to people's privacy, but to their personal security. Even when you can't demonstrate that a specific harm has occurred yet. It's like hazardous waste: given enough time, it will leak out and cause damage.