Glass is usually harmless, if you don't use lead or uranium glass. But breakage will lead to harmful shards, which is why many locations such as some schools or party places ban glass bottles.

For storage, use glass bottles. The material of the seal is usually not a problem if there is no contact to the liquid and the seal doesn't crumble. When reusing bottles, make sure to only use fresh or at least good-looking caps. And sterilize things properly (taking into account what the seal is suitable for), because mold and bacteria are far more harmful than common metal or plastic contamination.