Just because something is "synthetic" doesn't mean it's toxic. Plastics, or rather the additives used to make plastics soft, flexible, water-repellent, etc., are now known to have a specific set of toxic (or endocrine disrupting) effects on biological systems. Other synthetic substances may have none, or different ones. Textile dyes don't contain any endocrine disrupters as far as I know, but they certainly might have other ingredients that aren't very good for you. Also, some bioactive substances may be easily absorbed by the body, others not. There's a lot about all this that we just don't know.
Personally I'm not very worried about absorbing endocrine disrupters through the skin from clothes, but I avoid polyester clothes because a) I don't like the way they feel on my skin, and b) because of the environmental effects (washing them creates micro-plastic pollution, and eventually they become plastic trash).
But it should be pointed out that cotton is also an environmental villain... in cotton agriculture huge amounts of herbicides and pesticides are used, even more than for most other crops. And even if the dyes don't have any significant effect on the health of the wearer, the process of dying textiles produces lots of environmental toxins and probably isn't good for the health of the people working in that industry.
So in relation to the environment and polyester vs cotton you're damned if you do and damned if you don't. The one thing you can do is consume less... keep clothes longer, mend them whenever at all possible, or just go full naturist! ;-)