Occasionally, some anti-cheat software gets dissected and the analysis posted publicly. Here is an analysis of BattlEye with some information on its development history: https://vmcall.blog/battleye-anticheat-analysis-and-mitigati...

Typically, anti-cheat is separated into a frontend and a backend. The frontend becomes active at an enumerated set of events/criteria and collects data (screenshots, keystrokes, directory listings, process lists, results of memory scans, if instructed by the backend also memory contents or file contents) to the backend for further analysis.

These criteria are often ad-hoc (such as "if processes are running out of a user directory, capture them") and usually accumulate over time.

I would be interested in this, too. I read an interesting write-up by the guy who wrote the first .dll to hook functions in some FPS, I forget if it was quake or counterstrike. Sadly I can't seem to find it, now

Detailed discussion of how cheats and anti-cheats work is somewhat lacking as all the experts on either side of the issue would prefer to keep their secrets. These days an anti-cheat team that's serious about their job will lurk cheat forums and even try to poach talent there.