Personal preferences aside, they specifically talk about how to make the game "fun". For example, they try to account for things like real-world leader's personalities. "Gandhi" shouldn't just attack his neighbors unprovoked, but maybe "Caesar" should.
And why do the AIs even bother to ally/trade/help the player at all, since the player is the only enemy that really matters anyway? Because the player is also the only person that matters in terms of having fun, and making the game too hard isn't any fun.
At first I thought he seemed to be making a lot of false dichotomies, but really, he was saying, these are my only choices, given the overarching constraint on how much effort we can put into our AI. Basically, all of the AI decisions he talked about were small scale heuristics, with no ability to really look at the bigger picture, which is what the player is always doing when making small decisions.
And of course, personalities are a great idea, and not mutually exclusive with a strong AI. One thing I very much disagreed with was when he said an AI can roleplay, but a player wouldn't do that, the player just plays to win at all costs. But earlier on he even talked about one of the player types is people who play for a narrative. Those are roleplayers. So, especially at lower difficulty levels, it makes a lot of sense for the AI personalities to trump optimal play. But that doesn't mean you can't have a strong AI in general, that makes some limited sacrifices for the sake of fun.
Likewise with diplomacy. It isn't about making the game as difficult as possible. If that were the goal, you could just make the player lose when he loaded the game. It's about making it feel like you're on roughly even footing with your opponents, so that success has meaning. To that end, I would expect each empire to act in its own best interest, and therefore treat the player the same as it would any other opponent. It's also why diplomatic options need game mechanics to enforce them. As he pointed out, if there's no mechanic to stop a player from accepting money for peace, and then declaring war again, then there is no reason to every pay for peace. You should design these mechanics with player behavior in mind, and the AI should use them the same way players do. Think about how players behave with one another, make that fun, and then make the AI behave similarly. This wasn't possible given their constraints, but it's certainly possible in the more general sense.
As far as "making the game too hard isn't any fun" that's the problem. Too hard is subjective. What's too hard for one person is perfect for another, and way too easy for yet another. If anything, to please the maximum number of people, you'd have to write (or train) a very skilled AI that can compete with the brightest players. If you can do that, it should be much easier to tone it down or penalize it to make easier difficulties for the average and casual players.
What is your goal as a designer? To make a game with state of the art AI, or to make a game that has AI that keeps people buying and playing your game?
I know which I would choose.
Yes, 90%+ of players play on the single player setting - despite an awful AI. And I say that as one of that 90%. It is certainly not the AI that made me buy and play Civ. I don't expect nor want AI to play perfectly, but I do want AI to play good enough to not make ridiculously dumb decisions.
For example, notice how often AI leaves embarked units unprotected, so they can be easily destroyed. Or how often despite having long range artillery, it will enter town's firing range (even with that artillery!). Or how with its own towns it prefers so much to fire at even slightly damaged units, before attacking siege weapons (which deal much greater damage to towns).