As someone who hasn't studied this (yet) does that mean that the reason protons and electrons don't annihilate is because their Baryon Number wouldn't sum up to zero? (With Protons having 1 and Electrons 0 if I understood correctly)
If you start with a proton and an electron, you have 3 quarks and 1 lepton, so whatever the interaction you have to end up with 3 quarks and 1 lepton (plus any number of quark-antiquark and lepton-antilepton pairs). E.g. for beta decay:
neutron (3 quarks) -> proton (3 quarks) + electron (lepton) + antineutrino (antilepton)
But baryon number conservation is an asserted symmetry and there is no fundamental reason it holds. Finding proton decay would demonstrate that it does not but so far there is no evidence that protons (or bound neutrons) spontaneously decay.