freeciv is a noble effort but lacks features players want. For one, the AI is not on par with Civ 3.
I think it should be possible to make, not a good one, but certainly a much better one, with a combination of various classical game tree search techniques, it's just that AI doesn't seem to have been a huge priority for the developers. There are even mods for Civ 5 that drastically improve the vanilla AI just through tweaking various behaviours.
For graphics intensive games though, you either want something that can get by on the CPU or in future maybe TPUs will become common enough in consumer PCs so you could offload your AI onto that and still keep the GPU for fancy graphics.
At some point in the future I can imagine being able to plug in a pretrained alpha-go model into any game for fine tuning, that would be sweet.
It means the computer players will no longer role play as historical leaders but instead behave like a bunch of ruthless, psychopathic, cyborg Hitler clones. Most of the historically-informed trade and diplomacy options will be useless because the AIs will just betray you at the worst moment. In essence, they’ll play like human players and honestly, Civ games are just not designed for that. There are much better games out there (mainly board games but also RTSes) for playing against humans who play to win.
I say this not because I think it's an especially noteworthy or important objection, but to echo GP's point, that it's very hard to find AI that suits everyone, and it's not just a matter of difficulty.
And by having different AI at different difficulty levels, I think you actually can have AI that suits everyone. But I don't think most game companies like developing several separate versions of AI. They just want one that's superficially "good enough".
My friend and I, both launched a incredible min-max campaign against these guys, using nukes, and whatever fast units we had to conquer and raze cities that had spaceports to stop them. We managed this very quickly, it was like my own desert storm in terms of how quickly and well it went for me.
Then some random AI snagged a diplo victory and I was livid. I disable diplo victories when I play now.
I remember Brad Wardell claiming Galactic Civilizations would have the best AI possible, and it was certainly decent, but players still identified plenty of situations where the AI was being stupid. Brad said he did that intentionally because doing otherwise would be too mean. Thing is, hardcore players did use all of those tricks. At the highest difficulty level, the AI should too.
Use the reasonable, nice, historical AI for the middle difficulty levels, but allow players to choose AI that is just desperate to win.
For my current game, I took a step back to Emperor level, though. I wanted to try an early game Domination Victory, chose Alexander the Great on Pangea and Epic speed, and it's just way too easy (except for holding on to conquered cities during a Dark Age).
I think I'm going to try the same thing on Immortal next. Maybe even slower speed so we spend more time in the lower tech levels where all Alexander's advantages are. And I need to start spying earlier; stealing tech boosts helps a lot in keeping up. I'm still not quite decided whether it's better to stick to a regular age all the time or flip between dark age and heroic age; Heroic Age is awesome, but hard to guarantee, and holding on to conquered cities is very hard during a dark age (I just learned that the Bread & Games project exists primarily for that situation; it helps a lot to have an entertainment district near your front lines).
If you don't know the game well enough to win at the highest difficulty, it is not degrading to play at lower difficulties to learn the ropes while still enjoying the game.
Compare to Crusader Kings 3, which is nominally in the same general genre, but has rules that actively push you and the AIs into playing 'in character', because if you just ignore all the secondary stuff your attempts at conquest will almost certainly fall apart.
In this particular case that isn't the concern. Hitler has a single excellent strategic insight at the start of the war (movement wins, "Blitzkrieg"), but one reason the Nazis lose is that his later strategies don't work. Hitler really wanted to do Sealion (German ground invasion of England) after it became apparent Britain under Churchill will not surrender, but it's just not practical and his general staff persuaded him not to smash his face against that particular wall.
Sealion would be like a Civ AI repeatedly moving a just barely sufficient army from the French coast towards England, each time losing some to the water because they don't have good enough boats yet and each time discovering they can't be reinforced because that's too much movement - so the defenders have an advantage, and the advances all fail. The opponent is bloodied, but loses no territory while the AI's best troops are destroyed for no gain.
A Hitler clone, but perhaps not in the way you meant.
With imperfect information luck is a factor. An AI which focuses on luck outcomes might be more interesting to play against actually. Why have 100% chance to take exactly one of five enemy cities when you could have 20% chance for each tried in parallel. Less than 1% of the time you crush the opponent with all five defeats, much more often (but still less than half the time) you get none of them, but it always feels like you're a gambler this way.