This is why it's a known mutagen with a long list of very specific, strident toxicity warnings. That's the organic cation, though; compared to that, the Br- anion is utterly benign. You're looking at the wrong hazard!
Br- is a standard counterion in human-approved drugs (those formulated as HBr salts).
https://www.google.com/search?q=hydrobromide+site:drugbank.c...