Warmth is energy, and injecting energy into a large, chaotic system, like our planet's climate, certainly causes the average temperature to increase, but it also causes the system to become more chaotic.
More concretely, climate change also means more frequent hot spells, more frequent cold spells, more frequent flooding events, more frequent droughts, more frequent storms, etc.
Along with certain average temperature ranges, crops need many months of stable weather in order to usefully produce. Even though, on average, the temperatures in more areas will move into the range needed by various crops, the average amount of necessary weather stability will also decrease, in the potential new growing areas as well as the old.