I understand the concern, but also we do all have to act as ourselves in public life in many ways - any one of your identifiable actions could always come back to bite you.
If you want to build any kind of public profile or have much influence in life you kinda have to face this risk head-on to some extent.