Packets don't travel in straight lines.
However, ipinfo still appears to rely on active probing to triangulate geolocation data, which suggests they believe these routing asymmetries can be modeled or averaged out in practice.
The telco DSL and fiber in my metro area all runs through a single location where the PPPoE (hiss) concentrator is and the first hop latency from DSL interleaving swamps the latency from distance. You can someone is in the metro area, but not the county or city.
Cable company customers are a little more locatable, probably get the county.
Tier 2 ISPs have the right incentives to peer with every network in every peering point, so you are likely to get a correct result from a different ISP. Tier 1 ISPs avoid peering, so if your probe or target is on Deutsche Telekom or Cogent, your packet takes a circuitous route, passing only through networks that paid the extortion money.
Friends don't let friends get internet service from Deutsche Telekom or Cogent.
so theres funky overlap wherein on one isp you appear closer to city A, and on isp 2 closer to city B, but its same physical address.
Continental classification I'd think would be good as they appear to be coalesced endpoints, separated by vast oceans.