I admit it was more for the geek factor than anything else, and I'm no expert in the field. Here are the source stats[1] from chrony:

Pi with GPS

    $ chronyc sourcestats
    210 Number of sources = 7
    Name/IP Address            NP  NR  Span  Frequency  Freq Skew  Offset  Std Dev
    ==============================================================================
    PPS1                        8   6   112     -0.018      0.020   -105ns   336ns
    GPS1                        7   6    95    +55.685    111.018   +126ms  1058us
    GPS2                        8   6   112     -0.018      0.021    -89ns   341ns
    2a01:79a:408::250          18  10  310m     -0.057      0.080  -1364us   522us
    time.cloudflare.com        19  13  344m     -0.068      0.038  -1274us   258us
    ntp.coreless.net            6   3   85m     -0.343      0.706  -2355us   354us
    anduin.net                 21   8  379m     -0.020      0.051   -835us   388us

Debian VM running on Bhyve(FreeBSD) in same LAN

    $ chronyc sourcestats
    210 Number of sources = 5
    Name/IP Address            NP  NR  Span  Frequency  Freq Skew  Offset  Std Dev
    ==============================================================================
    timepi.localdomain         13   7   64m     +0.001      0.031   +328ns    28us
    79.161.39.253.static.lys>   8   6  120m     -0.002      0.184  -2122us   189us
    125.51-174-177.customer.>  22  13  464m     +0.287      0.117   -839us  1201us
    time.cloudflare.com         6   5   85m     +0.009      0.472  -1615us   277us
    c1EE4653E.static.as2116.>  10   7  155m     -0.146      0.603  -3315us  1256us

So on the VM it seems to "just" be an order of magnitude better. But that should be worst-case.

[1]: https://chrony.tuxfamily.org/doc/3.3/chronyc.html