Incorrect, mostly.
Your website is likely hosted on one or two servers in one spot on the Internet. To users physically local to that/those servers yes direct access would be faster, but what about people out of sate? What about those on a completely different continent?
Most of Google's services are hosted in several distinct locations and they use and their DNS setup makes sure that any given user gets sent to the one closest to them.
Even without DNS to help by handing out different IP addresses for the same service depending on the location you are calling from, they have an anycast routing arrangement whereby even when referencing some things by IP address directly you will get a local service of which there are several. For instance if I ping one of their public DNS servers from my home connection (in the UK) I get:
64 bytes from 8.8.8.8: icmp_seq=1 ttl=56 time=30.6 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=56 time=29.2 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=56 time=28.9 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=56 time=38.4 ms
64 bytes from 8.8.8.8: icmp_seq=5 ttl=56 time=36.5 ms
and when doing the same from a VM running state-side I get: 64 bytes from 8.8.8.8: icmp_seq=1 ttl=56 time=12.3 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=56 time=12.3 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=56 time=12.4 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=56 time=12.3 ms
64 bytes from 8.8.8.8: icmp_seq=5 ttl=56 time=12.3 ms
It is simply not possible, unless packets from/to one of those locations are travelling faster than light speed and/or nipping back in time during their journey, for these two locations to have been talking to the same server(s) despite using a specific address not a name.As well as locality of reference, there is the benefit that your most local Google site is likely to be connected to the backbone by a faster pipe than your hosted servers (particularly for users of large ISPs who have more direct peering arrangements with Google). This comes into effect for any static content, or dynamic content where you can tell Google's checker when the content has not changed so the last version can be used, where the user does not need to wait for your server to respond in full to Google
For truly dynamic content this is going to be slower, but for many sites, especially if you have proper cache control on your dynamic content, I would expect this service to produce a measurable speed-up on average (obviously you'd have to test your specific site to see how much difference it truly makes) and it could save you a fair chunk of bandwidth if the site becomes popular.
This is certainly a service I'll keep my eye on with a view to maybe making use of it for certain projects if they ever get off the ground.