That's a bit of an oversimplification that makes this sound more concerning than it is. Not that it's 100% un-concerning, but I suggest that anyone concerned about this read the linked docs. It really was designed to be quite privacy preserving, though it's obviously a difficult challenge in this context.
The face alone that if you spend 10 minutes within someone's proximity will get flagged is 100% concerning. Of course they keys will only be handed to government health officials, but we all know what that means.
Keeping the name out of that data does a lot for privacy. With a name in there, you could use the data to find my name and figure out my address. Without the name in there, you don't know which point I even am without knowing some of my locations from another source.
The contact tracing APIs very deliberately avoided GPS for exactly the reason you mention.