I'm not disagreeing with what you are saying, but I think the issue at hand is very different than what you are arguing about and must be taken in context of the greater game design: past CIV 4, the game design got disconnected and ran ahead of the AI. I'd imagine that the AI in the later games is actually much more advanced, but it feels incredibly dumb. Truly catastrophically dumb, because the game mechanics were not made in mind with the capabilities of the AI, or the AI was not made smart enough to function with the game mechanics. I'd argue the latter is much more difficult to do (and here we agree), but it doesn't excuse the development team.

Take one of the most controversial features of CIV 5: one unit per tile. On the face of it, this is a great feature that opens up interesting tactical opportunities. Blockading, the importance of terrain, formations etc.

Unfortunately, whether due to unwillingness or lack of competence, the AI was completely befuddled by this. It'd make big armies, and then be fundamentally incapable of moving them. It would shuffle units one by one around endlessly and get stuck in the smallest of congestions.

This is not about whether an AI should play to win like a human or it should roleplay. I agree - and the majority of playerbase agrees - that the CIV AI should primarily roleplay. We all remember the frustrating double-crossing bastards during the early days of CIV 5.

This is not about the AI being complex or not. This is about making a game where the AI is competent, and this is not just the AI being some super smart intelligence in absolute terms. It is (mostly) about designing the game such that the AI can appear competent and pose a threat where the human player can feel like they are playing against an opponent that can test their skill, instead of just trying to beat some pre-determined math that is stacked against them.

CIV 4 was a game built in an era where computers were orders of magnitude less capable, and yet the AI in the game poses a much better and more competent appearing threat than in the current iterations. Not because it was more advanced, but because the game was built such that the AI didn't need to be that advanced. The AI still cheated of course, but it only needed a few simple heuristics and rules to build doomstacks that appeared "smart". A simple A* was enough for it to navigate this doomstack to your capital and make you sweat.

Now we have very complex tactical battles with positioning and formations, but this would require vastly more complex AI to do well. Unfortunately, the AI has not kept up, so we are left with opponents that appear dumb as bricks because they don't play the game the human plays. The AI simply spawns resources and armies into existence out of thin air in ever more blatant ways. The less the AI plays by the same rules the player has to, the less it feels like a story and more like a math puzzle.