https://en.wikipedia.org/wiki/Urushiol-induced_contact_derma...
and it notes that
> People vary greatly in their sensitivity to urushiol. In approximately 15% to 30% of people, urushiol does not trigger an immune system response, while at least 25% of people have a very strong immune response resulting in severe symptoms.
I've also heard this about poison ivy. (I assume cashew plants have less of this irritant than poison ivy does.)
But it also varies a lot among people. My wife has no issues eating/handling mangoes but her twin sister has horrible allergic reactions.
Often subsequent reactions are worse than the initial.
Like most allergens, people have very different reactions to it and can grow tolerance over time with exposure.