I'm not sure I'd classify most of these as AI agents, it's mostly web crawlers. (Though the definition of agent is fuzzy enough that I suppose you could lump them in).

I'm also curious, will adding "Common Crawl" to the user-agent disallow list in your robots.txt actually do anything?

Added context: I run a tracker of vetted AI agents with verticalized use cases https://staf.ai