One-off ingestion of metallic mercury isn't a big deal, it basically comes out the other end. But mercury fillings continuously emit a tiny amount of mercury vapor which the person will inhale continuously. Very small amounts but measurable.

At the end of the day there's massive amounts of data since these fillings have been in use for over 100 years, and it's quite clear who has them and who doesn't. If they do have actual negative effects, they are small.

Definitely better than having cavities, but modern composite fillings are probably better still (though, being plastic, there's the endocrine disruptor question on those - and a much wider range of composite filling materials out there makes it harder to study).