To give a denim example:
Cotton growing wrecks the land, farmers gets peanuts.
I'll give a pass on the actual fabric making and sewing process, but what really got me was the amount of energy and robots used to destroy denim to make "distressed" jeans.
In some processes you have many hours of fabric being distressed, in wheels, in drums filled with different types of pebbles, giant presses, even lasers to slice them up... the jeans spent more energy and way more time getting ready for fashion then they did getting produced. (not considering cotton growing time).
All this work to produce something which lasts a lot less then it could, and probably will get thrown away soon. Clothes need an environmental impact label.
I have a new found respect for man-made fibres like polyester, polypropylene etc as producing those fibres is so much cleaner than natural goods, and instead of having to spend a lot of energy getting the fabric to "feel" right, you just modify the properties of the fibre by changing how its produced.
Edit: Fashion is one of the few remaining industries where obsolutely zero consideration is given to environmental impact when designing and producing clothes. This isn't just a poor country thing where they make clothes in a enviornmentally destructive way, the design of the clothes involves IMHO a lot of un-necessary production steps for an extra minuscle percentage of improvements.
Other industries, like say car parts, think about things like the energy inputs to their production process from the design state onwards, fashion doesn't. Even in sevice industries like making web apps, people think about inputs, and design to use as few of them as possible.