The safety deposit box company was pretty clearly a criminal enterprise, it marketed itself to drug dealers and so on. A few random rich people might have wandered in but it was a fairly unique company.
The point here is that this shouldn’t fucking matter. The government should have to prove its case anyways by having individualized probable cause and a warrant for each and every safe deposit box, and going on a fishing expedition is patently unconstitutional.