But you need only small amounts of it. In a LFTR, one ton of thorium, about the size of a beach ball, is sufficient fuel for a one-gigawatt reactor to run for a year.
Compare that to the enormous quantities of coal ash we produce every day. It's hard to imagine chemical toxicity being a major problem. People used to make lamps out of the stuff.
In fact, given that thorium is a byproduct of rare earth mines, and currently considered a hazardous waste, it'd be nice to have something to do with it.