> I know he wasn't popular in the US which I don't really understand.
Note, these are not my opinions, but I am going to seek to answer this based on the things I have read on him.
In short, the general opinion is that he was a weak, pitiful, activist president. His overtures toward peace, anti-imperialism, humanitarian causes were laden with activism rhetoric that one would normally use if they were outside the structures of power. Once one is within the structure of power, then one is expected to speak less, and do more. Consider a parallel story of outspoken-activist-turned-president Theodore Roosevelt, speak softly and carry a big stick and all that. Carter spoke loudly and forgot his stick. This, combined with a perceived lack of policy and agenda, on top of a time of various crises, made him look rather unfit for the job.
Again, please remember that this is the summary of a general stance many hold. I was not yet alive and so cannot necessarily speak to the zeitgeist of the time, firsthand.
My personal opinion is that, if you squint, his presidency is a lot like that of Trump, in that his presidency lacked a focus and tried, and rather failed, to react to events at the time. And of course, rather different in that he was not a corrupt buffoon. To expand on this, I actually think that if we waved a magic what-if wand, and got Carter today, he would be hailed as a fantastic president, if only by comparison, so you know, context matters.