I did say this: If the consumption of a particular food can be shown to have a net positive impact on one's health and longevity, and the properties that generate that positive impact can be distilled and consumed, it behooves us to do so.
The evidence I have seen overwhelmingly states that taking supplements has little to no measurable effect in people with a sufficient diet. The reason that I wouldn't add it to my diet is the same reason that I don't take oxygen supplements -- as long as you generally have enough oxygen, there's just not a whole lot of evidence that you need MORE.
So, youre right -- chances are that there's not reason to NOT take them. But there's not really a reason to take supplements, unless you've identified yourself in an at-risk population. But those groups are few and far between compared to the number of people who take nutritional supplements.
Then again, I'd rather save my money than buy the modern equivalent of snake oil tonics.