Nobody is going to invest the resources into a real AI just for the hardest difficulty modes. If it cannot be used on Easy/Normal, it simply isn't happening.
Which is really sad because "hard" difficulties in most games just mean that you have less health and NPCs do more damage or their "cheat factors" are turned to 11. They're still the same stupid NPCs as on "easy", but now they just headshot you behind cover and have zero recoil / perfect accuracy. It's technically difficult but not really fun.
One of the most obvious examples of this is GTA 5 where all NPCs are equally dumb, but "harder NPCs" just mean that they tank huge amounts of damage, while they have very little aim delay and essentially perfect accuracy. Due to various bugs in the game they'll often shoot through walls or even entire buildings: technically hard, but not fun.
Which makes no sense at all.
Luckily for Skyrim the game thrives on mods anyway.
Market size is only half the equation, the other half is the cost to develop the feature. As methods and technology improve, the cost to make a smart AI for a complex game system should decrease. As the cost decreases, you need less of the overall gaming market to make a return. Might just be a matter of time before somebody takes a gamble on a tough AI, has a huge success, and then everyone starts to emulate it.
Smart AI is easier to make than a mediocre, fun AI. Optimality is is a clear goal. Fun AI is less clear and requires a unique blend of technical and design expertise.
In a real time strategy, it seems we're approaching that level with that StarCraft ai. Need to perhaps restrain the ai's ability to micro by fuzzing it's ability to control quickly and precisely in bursts, and then continue to advance it's strategic ability, but I expect it'll get there eventually.
In the case of shooters. It might involve giving the AI a noisy view of the game world, to force it to approach the problem the way a human does. Humans don't just have an accuracy rate or a timer between an event entering their screen and their ability to shoot at it. A human has to filter through visual noise to notice a target, and decide how to engage. A human has to use an arm and hand and muscle memory to manipulate a mouse to get bullets on target. Those are all imperfect controls, with a certain amount of uncertainty between intent and action. All that is further influenced by how prepared the human was, how much they have to adjust their previous plans, or recenter their attention.
It's a difficult task to identify all the things that actually make a game difficult for a human and make sure the AI faces a close approximation of the same problems. Then it's even more difficult to get an AI to a point where it can compete on a human level, not to mention various human levels.
That doesn't mean it's not possible or not worthwhile.
Like I said, though, many players choose a goal oriented play through. Achieving that goal is a win for them (and often they abandon the fortress and start a new world afterwards). I remember one person bitterly complaining on the forums that the game was too easy because they had already won the game. After they succeeded in becoming the mountain home for their civ, they felt the game was over. Sometimes it's hard to explain other ways to play games ;-)