No, but there is a pretty small set of people who can access the information. It's not like anyone with a grudge can link a Bitcoin address to a real life person.
> The address isn't tied to you, so how would anyone figure it out?
Or
> there is a pretty small set of people who can access the information.