It took me a long time to figure out why these things happened, by checking ingredients of everything, finding what's common, eliminating elements until I've reached the one that caused the issues. That was aspartame.
The fact that it may cause cancer really is enough to avoid it. Smoking three packs per day is more likely to going to cause you cancer earlier than smoking only five cigarettes per day, yet people don't run around claiming that it's fine because it's only so few anyway.
It's really problematic that societies have adopted this idea that a little poison is totally fine, as if there wasn't already hundreds of little poisons and damaging effects acting on your body on a daily basis.