Firefox and Chrome use DNS over HTTPS by default I believe, which may mean they use a different name resolution path.
The above is entirely conjection on my part, but the guess is heavily informed by the surprise of curl's behavior.
But operating system resolver only speak with DNS servers. It does not make https connections to calculate latency which would pick "the closest server". Also dns had no way to tell what port you will be using, maybe service is on 8443 or something.
For geo DNS I've built a custom backed for powerdns with geo DNS capabilities and healthckecks to quickly remove a broken vps from the DNS responses.
No way MacOS parse tls clienthello looking for SNI.
Also I doubt a DNS resolver runs in the Mac kernel, ring 0 to pull this off.
The thing with DNS is that it works on layer 3. Hold on, what? Yes, layer 3 because you obtain network address for layer 3 (ip4, ipv6) but latency can be measured only in layer4 (tcp, quic). Of course I know that common wisdom says DNS is a layer 7 but from functional perspective, you are yet to establish your destination network address, therefore functionally it's like layer 3 to me. Or even lower, because without destination, you can't even start creating a packet and inspecting your routing table entries figuring out if you can even reach it ;)
There is zero chance Mac resolver libraries can connect you to the fastest responding server - unless there is no Berkeley sockets but something that allows you to do a connect(char * fqdn) and system library return you two pipes, one for write, other for read, and that you can close them independently. I doubt it there is such a thing, but don't know Mac os API.