The biggest pest I've seen personally has been flies. Tanchid flies will lay their egg inside the caterpillar and the larve eats the caterpillars from the inside and they die. So we round up caterpillars we see on our outdoor plants and place them in a protected mesh enclosure with potted milkweed for them to eat.
In 2021 I successfully raised 81 monarch caterpillars to full grown butterflies. In 22/23 we still had some success but I didn't have a garden so we raised 10-20 wild ones. But 2024 we have a house now and a big garden full of milkweed!
Make sure to plant native milkweed in your area!