His reply was in the context of you labeling yourself an atheist. Perhaps you should avoid that term in the future.
1. 'baggage' is not a synonym for 'rights' (legal? social?) 2. While an Atheist is necessarily a non-believer, a non-believer is not necessarily an Atheist.
Furthermore, you tell people you're an Atheist specifically because it conveys a lot with a little. Some of what it conveys is definitely not secular humanism, if that's how you prefer being viewed, you should start with that.
Not only is that not an offensive observation, it should be common sense.