I think of poisoning as more systemic, less topical. Capsaicin acts as soon as it touches something.

I don't know if it enters the bloodstream or not, but since people eat it without more than a surface irritation, I would think that a quantity enough to cause systemic damage would be far too concentrated to touch, much less eat.