You're right that it's not a true cold GPS signal. I've connected to the network in the last 180 days (and thus have information on satellite information), I have a rough clock (time within 20s or so). The phone probably assumes it's closeish to where it last turned on. Etc.

It is the realistic scenario that people would face without using WiFi routers or cell towers though. True cold fixes aren't something phones need to deal with basically ever.

It doesn't need to be a cold GPS signal, it only needs to be one that doesn't track you. Your drone doesn't have access to the internet to do the almanac and ephemeris download, that's the difference.

Aiding the fix by obtaining data from a faster unidirectional source (instead of satellite to device, it's just internet to device) doesn't reveal anything private, unlike mass-capture of WiFi (B)SSIDs.

Also, I would look into the quality of the GPS radio and antenna you are using - $95 u-blox modules will get an RTK-quality fix without outside assistance in minutes.

20minutes just means a shitty module or lots of ~1.3ghz noise (e.g. from dji o3/o4 or gopro cpu), although bad input filtering and gps located near an antenna could saturate the amplifier even with antenna on a different frequency. 1min is typical for cold fixes but you can make it faster by uploading ephemeris/cache using AssistNow (valid for about a month).