Do you not sync to iCloud?
If only you and three other people have a unique file, Apple knows you are a group.
Most people broadcast their lives on social media, happily opt in to all sorts of schemes that track them just for minor conveniences. For people like that, the idea that the privacy protection outlined by the OP isn't enough rings really hollow.
Or to put it bluntly, at some point this really stops feeling like a practical debate, and more of an ideological one.