If it doesn’t make sense for people to say “vegan except for chicken”, “vegan except for beef”, then why on earth would it make sense to say “vegan except fish”?
Fish aren’t lesser beings, they are living creatures like anything else.
Just stop saying “vegan except fish”. Say you eat plants and fish. That’s all you have to do.