ipinfo.io uses a probe network for this[1], but even then a server physically located in the Netherlands with an IP announced as being from, say, Seychelles would still respond to pings faster from a European location than from somewhere like Singapore (unless you go out of your way to induce latency to ICMP responses).
[1] https://ipinfo.io/blog/probe-network-how-we-make-sure-our-da...