I'm more concerned about plastic fibers getting into the air via laundering (or simply regular activity of wearing them and little bits of plastic occasionally coming loose) than I am about the use of oil in the clothing.
I believe there should be more restrictions on the use of plastic in society, seeing as "Plastic Fibers Are Found in 83% of the World's Tap Water, a New Study Reveals"[1]