I don't know the quantitative answers to any of these. But I would hazard that wood decays fast; and glass and concrete particles are inert, very similar to soil and not very harmful in the amounts they make into our food supplies.
Plastics on the other hand, don't decay, float in water (so easier to spread) and empirically make it into our food supplies. The fear is that they are carcinogenic and cause unwanted hormonal changes.
I know it sounds crazy, but tupperware is a thing. We had ways to distribute, dairy, meat, and all types of food before plastic. But even if we do make use of plastic in the future, we need to rethink how distribution happens instead of just slapping everything in its own plastic bag/box for its entire lifetime. It is entirely thinkable that you bring your own tupperware to the grocery store and fill it up, then bring it home. I've been doing this for years with takeout. People are just lazy and can't plan ahead.
Progress is about making good tradeoffs, not about picking perfect solutions. Otherwise your logic could get rid of any innovation by ignoring its benefits and saying "it has some problems so let's get rid of it altogether, we somehow survived without it in the past".
These things (including plastic) all have widespread costs and widespread benefits. Let's try to find ways to reduce the costs without entirely ignoring and losing the benefits?
60 years ago we were putting way dumber contaminants in our bodies.
The paint on your walls is plastic, as are your sewer pipes, probably water pipes, the insulation in your walls, the glue such makes up structural members of your house...