For storage, use glass bottles. The material of the seal is usually not a problem if there is no contact to the liquid and the seal doesn't crumble. When reusing bottles, make sure to only use fresh or at least good-looking caps. And sterilize things properly (taking into account what the seal is suitable for), because mold and bacteria are far more harmful than common metal or plastic contamination.
Also, corks promote microbiological contamination of various kinds, leading to e.g. the common failings in wine (there is actually a multitude of "cork" faults, not just one): https://en.wikipedia.org/wiki/Wine_fault
I'd prefer any metal or plastic to cork actually. Just because it is "natural" or "traditional" doesn't make it any good.