Lead, mercury, fluoride. Other cumulative neurotoxins. It seems obvious to me that we should work to eliminate anything which damages our brains, whether or not we can test yet for effects.
Even though the research on the effects on the brain is inconclusive, you probably should avoid ingesting too much fluoride to avoid skeletal fluorosis. And if you take a large dose of several grams, it drains your blood of calcium, which the nervous system really doesn't like.
[1]https://en.wikipedia.org/wiki/Water_fluoridation_controversy
Part of the problem of course is politics. Dental health and vaccination are both very important good things and this makes it even harder to determine the truth of the matter. For example, people regularly exclaim things like 'there's absolutely no evidence that X', or 'there's overwhelming evidence that Y'. But they do this rhetorically, and as I said, without associated explanations. In such circumstances 'evidence' isn't always meaningful.
Hence my original point. Since we don't yet understand how the brain and the mind work (if we did, we'd have AGI) we're not in a position to assess damaging effects, if any, which may be subtle. If this was any other organ, I'd say let it go. But the brain is the seat of the mind and is therefore uniquely precious.