If a site uses your Gravatar, game over: Gravatar's literally a raw MD5 of your email with the aim to give you a globally identifiable avatar. That's been known (documented!) for a long time and unlikely to be a surprise to many of us here.
The only place this is likely to be an issue is when a site knows you but you assume you're anonymous. If you're using a web service where you want to be anonymous, connecting anything with your identity is a bad idea.