It's bigger than that.
So first off, just about all of these artificial sweeteners are like 200x sweeter than sugar or higher. This kind of means that you have high amplification which comes with distortion—artificial sweeteners do not taste like the real thing. So those chemical aftertastes are probably the biggest contributors to skepticism.
Second, the most common artificial sweetener right now is aspartame, and it is kinda nasty stuff. Don't take my word for it, drink a Diet Coke that's about 1 week before expiration and see if you can stomach it—it's foul. That foulness is the metabolites of aspartame, I want to say one of them is formaldehyde? They are pretty nasty and you do produce them after drinking a Diet Coke, they appear in urine afterwards. The aspartame also bears a cryptic warning for people with phenylketonuria, and it might legit make consumers feel worse, as in reduce their serotonin and dopamine and make them more irritable.
The only silver lining on this cloud is that because this stuff is 200 times sweeter than sugar, you actually only get a very low dosage per each individual beverage. Because of that very low dosage, we are thankfully not seeing huge epidemiological problems from it yet. But yeah, a combination of nothing tasting right and the poster-child being awful is a decent recipe for conspiracy theories.