Back in the '90s, people would say "oh he's 'special'" as a slur.
That being said, in a world like we live in today, pretty much anything you say or do will offend someone for some obscure reason that you just can't reasonably anticipate.
It's in these contexts that I think the most appropriate response is "get a life."
Eh, this is a very particular and long-standing racist term, and the meaning used by the authors is derived from the slur, so it's not incidental.
There's also rather significant modern use of "tinker": https://en.wikipedia.org/wiki/Tinker_(disambiguation)
It appears that you're getting stuck on "tinkers", which is mentioned in the above disambiguation page:
> Tinkers, an alternate (and often pejorative) name
Anyway, language is malleable and changes. In this case, I will again emphasize "get a life."