My phone doesn't take several minutes to get a GPS fix even when I'm in the middle of nowhere well beyond the range of cell towers let alone wifi routers...
I understand that in theory it could if it had no clue what time it was either for some reason... but in practice we're talking seconds (though more than 2) not minutes.
It is the realistic scenario that people would face without using WiFi routers or cell towers though. True cold fixes aren't something phones need to deal with basically ever.
Aiding the fix by obtaining data from a faster unidirectional source (instead of satellite to device, it's just internet to device) doesn't reveal anything private, unlike mass-capture of WiFi (B)SSIDs.
Also, I would look into the quality of the GPS radio and antenna you are using - $95 u-blox modules will get an RTK-quality fix without outside assistance in minutes.