In my meager attempts at a backyard garden, I have lost entire "crops" of vegetables due to insects. From the slugs eating the young/new growth, grasshoppers eating the leafy plants, birds/insects eating the fruit, and just neighborhood stray cats using the loose soil as litter box digging up the plants. Then there's the squirrels and possums or raccoons that come along digging up plants looking for food. I just want to nuke the whole place from orbit. It's the only way to be sure.
With all of that, it's just a hobby to "keep me off the streets", but damn those pesky pests. I have often said that I'm really lucky/thankful that my livelihood does not depend on me being able to successfully grow food. Oh, and before you get to the stage of pests, it has to rain just the right amount. Not enough, and nothing grows. Too much, and everything drowns and rots. None of these things are in the farmer's control.