If you follow me, you tell my my server that you want my posts to be delivered to you.
That means my server has to know your account exists.
I also have to know that you follow me because I can export my followers if I move to a new service.
My server can choose to publicise the relationship. So anyone looking through my social graph will see "fsckboy follows edent".
By default, I believe the current Mastodon software does show that relationship - but only to me, people on my instance, and people on your instance. No one else. I'm happy to be corrected on that though.
In theory, an attacker could scrape the social graph of every user in the entire fediverse and build a list of who you follow from that. I don't think that's a realistic threat.