If I have your public key I could probe a server to see if it recognizes that public key. Which would tell me information you might not want to be leaked.
Could you explain how/why that would be an issue? I still don't understand. Isn't the nature of public keys meant to be that they're public?
I think what the other comment is getting at is that you're essentially leaving footprints behind on various servers and that information could be used to narrow down who you are, where you work, etc. For example, if you used the same key pair for a work machine at the big social media site example.com as you did late.sh, they could probe example.com's machine and see that you'd been there.
That sounds like the kind of edge-case thing that wouldn't be even slightly a problem for like 99.9% of people.