Yes, you only get the public keys of machines you can talk to. Later the server asks the client to forget the key if they can no longer talk.

Additionally, we rotate keys. For security sensitive users we can rotate these keys daily.