There are some very vocal people who argue that the risk of toxicity is very low. The studies aren't great. But for most people, taking a moderate dose of Vitamin D will be safe.
Unfortunately, I have treated many patients who come to me with high calcium levels due to taking Vitamin D. They do not have parathyroid disease (I have treated thousands of patients with parathyroid disease). They have simply been taking high doses of Vitamin D. High dose is generally 5000 units a day or more, though I have seen it occur with lower doses if taken over many years. Vitamin D is fat soluble, so it will stay in your system for a long time, building up.
Of note, some of those patients were instructed by their doctors to take those high doses. Even doctors forget the relationship between calcium and Vitamin D, and I have spoken with doctors who did not realize that 5000 units a day could cause problems. As a doctor, there is a lot to know and remember, and most doctors haven't studied calcium metabolism in many years.