Civ games have never been known for their stellar AI. The way they keep the AI competitive at higher levels is to brutally cheat by getting free techs and units at the start and a beneficial modifier to their production and research.
Considering how quick the AI needs to run (you don't want "end turn" taking forever) and how complex the game is I can appreciate its simplicity, it is disappointing though.
I have examined the sources for the Civ IV AI and must say that it isn't very pretty (hardly any comments too!). One thing it doesn't seem to do that often though is cheat by examining what other players are doing (such as not building wonders it wont get) which is something at least.
But still.... can anyone point me towards a game where the devs actually spent a significant amount of time (relatively speaking) on the AI? It always seems like such an afterthought. Mods seem to always be a better bet. Although perhaps that's because the most efficient ways to play a game are always developed long after the game is actually released.