Then you haven't been paying attention. It's not about growing bananas where today we grow potatoes, it's about what happens in the areas where today we grow bananas. Those areas will become literally uninhabitable for large parts of the year in the next 50-100 years at the current rates of carbon emission, and the billion+ people living in those areas will move elsewhere.
Looking at how the minute amount of migrants from Libya and Syria were received in some of the wealthiest countries in the world (~1 million people reaching Europe and throwing it into a political crisis), the effect of 3 orders of magnitude more migration will be devastating, and are one of the main reasons for imagining that nuclear war you fear (especially given India, Pakistan, and China have nuclear weapons, and will be at the forefront of this massive migration from Bangladesh and their own coastal areas).
All this will happen, most likely even if we reduce emissions to 0 today. Perhaps it will be limited to a few hundred million instead of more than a billion if we reach some goals of the Paris agreement, but even that is unlikely to actually happen.