Thorium is not much radioactive indeed, but radon it emits is. That's why f.ex. building houses on a granit (that is generally thorium rich) without proper ventilation underneath can result in serious health issues.
At the levels where it is recommended that you remediate your home, your likelihood of dying from lung cancer is equivalent to dying in a car accident. That's potentially 50 people a year when you look at a population of 200,000 people.