My understanding of arsenic poisoning is that it binds to some proteins that inhibit some processes in cells. This means it eventually gets cleared from the system. So as long as you don't get a lethal dose it shouldn't cause long term damage. There is arsenic in apples seeds and I've never heard of long term issues from eating a few of those.

Lead on the other hand builds up in your system and causes permanent damage.