Ester bonds are not usually dangerous. Wait until you find out what all the natural and synthetic flavorants inside your food are!
As far as polyesters go, the main cause for concern is the endocrine disrupting effect of polyethylene terephthalate[1] and the inability of PET to be broken down completely, leading to microplastic pollution. By contrast, PLA can be broken down by heat or a select number of micro-organisms, and the lactic acid monomer used to produce it is non-toxic. The polymer is formed by a simple condensation reaction to form ester linkages, releasing water. Esters are a very common and generally benign moiety found everywhere in nature, for example in cutin found in plants, forming the rigid exterior of cells exposed to the elements. [2]
[1] https://en.wikipedia.org/wiki/Polyethylene_terephthalate