Subset Games, the makers of FTL and 'Into the Breach' have talked about this extensively. 'Into the Breach' deliberately has the enemies telegraph their plans one turn ahead of time, and the game is all about interfering with those plans.
The rest of the game's design carefully reinforces the message that the enemy units are not intelligent. The backstory has them as basically oversized insects.
If you have a game that pretends to give you realistic human antagonists, but they behave mechanically dumb and predictable, that breaks immersion like you suggest.
One big problem is that having very smart AI that's purely there to oppose you in a zero sum game ain't fun to play against for most people. The handicaps a modern chess or Go engine would have to give you a normal human for a fair fight are ludicrous. And people seldom want fair fights in their games. They want a feeling of accomplishment, but without actually putting in all that much work.
Even hardcore games like XCom cheat in your favour behind the scenes.
There are at least two ways out of this while still avoiding the boring repetition:
- carefully make the NPC make believable human-like (or animal-like) mistakes, instead of easily exploitable repetitive mistakes - give the NPC goals that are in conflict with the player, but not 100% so.
A silly example of the second option:
Take a game like Thief that's all about sneaking around stealthily and stealing stuff. Now realistically, most of the guards are just hired goons. They don't want to die, but they don't particularly care about protecting the place. They do care about being seen doing their job, so they don't get fired.
So your job as a player could be, in addition to staying unseen, to provide plausible distractions and reasons for the guards not too investigate to closely.
Higher ranked guards, and owners, would be under higher pressure to perform and won't get away with excuses. So they would be more alert.
Using the same trick over and over again would lower it's effectivity: guards can't plausible claim to their higher ups to have been tricked again and again.
If you are starting to become aggressive to a guard, or he learns that you called one of his friends, the guard's priorities will change towards more self-preservation.
A pacifist run might even earn you respect and admiration from the lower level guards. Just like cat burglars are often admired in real life.
Seen a bit more abstract, the game now becomes one of three factions: the thief (that's you), the low level guards and their employers. All with partially overlapping, partially conflicting goals.
You can throw insurance companies into the mix, if you want to make it even more complicated.
Thanks to the non-zero sum nature of the partial conflict, you can crank up how smart everyone acts, without overwhelming the player:
Eg smarter guards might figure out a way to de-escalate that still looks like a plausible and even courageous move to their employers.
As an aside, unrelated to your reply, I just remembered another common excuse, which is that the enemies only exist for X seconds before the player shoots them, so effort on smarts would be wasted on them. For some games, I think its for sure the case, but I also feel that in many cases the enemy only exists for a few seconds because they are dumb and uninteresting.
Both 'dumb' and 'interesting' AI can be useful in a game, it all depends on your game's design.
To give another silly example: Tetris by default has very 'dumb' AI that just gives you pieces at random. You could imagine variants of Tetris with more interesting piece selection.
For example, an AI that makes pieces as unhelpful as possible while keeping their distribution statistically indistinguishable from true random selection.
Or there was an inversion of the 2048 game, where an AI plays the normal game, and your task is to give them unhelpful numbers.