Even these new, who-knows-what additive laced straws start falling apart after 5-10 minutes.
Maybe wax-coated paper, or a thin foil coating on paper, or some other kind of plant matter pounded into the right shape, or something.
Wax is a hydrocarbon that I believe comes from oil. I don't know how it breaks down on the environment but it's not immediately clear it's actually "better" than plastic, other than maybe not being seen once it falls apart.