Intelligence requires bigger brains, which comes at a cost - huge energy requirement (20% of total for humans), weight (for birds), head size (issue for human birth), etc.

Many animals have no need for intelligence/generality since they have a very limited behavioral niche (e.g. herbivores, crocodiles, sharks), so it wouldn't have evolved in the first place, but even for those that do the benefit has to outweigh the cost.

If every animal was a generalist they they'd all be in competition with each other, so I'd expect if you ran simulations you'd find that an ecosystem full of species that don't compete head-on is more stable, and therefore likely to result.

Inteligence is the endgame answer, brute forcing it gets you 85% of the way, but only inteligence will get you to 99.9%. Sometimes the local maximum is enough.