Agriculture is difficult because it is literally a fight against nature/entropy. And it still has to be proven that it is a viable way to drive human civilization in the long term.
In nature, species populations compete with each other and are kept in a dynamic equilibrium. If population size of a species increases, then its predators will have more opportunities to predate, and their population size increases as well. This leads to more predation pressure, which should negate the population increase and decreases opportunities for predators as well. "Predators" is to be understood in a very general way. It can also apply to fungi and other microbial forms of life.
Changes to this equilibrium arise when prey species evolve to resist the predation pressure by predators. And vice versa if the predators improve. Then, ecosystems change, and these changes can be very disruptive.
Agriculture is a rebellion against this dynamic equilibrium because it creates irresistible opportunities for predators. They have to be held in check either by brute force (hunting, pesticides, containment) or by hacking the ecosystem (for lack of a better term) to ensure that helpful predators are around. The former is probably unsustainable in the long term as it is not at all clear that we will forever be able to outsmart the evolutionary potential of predator species. The latter is difficult as well.
Apart from predation, the other challenge to agriculture is that it depletes the soil in the long term. Fertile soil arises as a result of a ecosystem where the decomposers cannot keep up. Most forms of agriculture interrupt such ecosystems and thus deplete the soil in various ways.
To keep this in check, different methods were evolved, such as slash-and-burn, crop rotation and artificial fertilization. But it is not at all sure that we will always be able to stop depletion of our soils just in time to prevent civilizational collapse. Currently, availability of phosphorus is a looming danger because most of it is mined. Once we hit peak phosphorus, we better have solutions in place to get it from somewhere else.