* You can set a read watchpoint using debugging APIs (ptrace);
* You could place the honeypot in a memory page(s) that has its read permission revoked. An attempt to read the page(s) causes a signal to fire. In order to not crash the application, the code would then handle the signal by making the mapping readable, before continuing execution as normal.
Other approaches probably exist too, these are just the two options I would personally try first.
So, e.g., you alloc a blank page into memory: it isn't mapped yet, so the first read will trigger a page fault. You register that page with your userfaultfd. You (Dota, here) never read from it. If the userfaultfd receives an even that the page is faulting, then it isn't Dota/you that's reading from it.
Judging from the comments it sounds like Windows has similar capabilities.
… there are all sorts of false-positives here. (Or with any honeypot, really.) Many are mentioned elsewhere in the comments…
(Cf., userfaultfd(2).)
What kind of read is sufficient to trigger this? If dota makes a read watchpoint with ptrace, my cheat process calls the linux equivalent of readprocessmemory on the dota process, then dota gets notified by the kernel? So every time a process directly interacts with the memory of another process, the kernel has to look through a list of which processes have called ptrace and run some kind of handler? As an aside it seems like this would be bad for performance of the whole OS
If ptrace is a syscall and ptrace (according to wikipedia) allows one program to intercept and manipulate another program's syscalls, then couldn't I just launch my cheat first, have it ptrace dota, and intercept dota's ptrace call, so that the read watchpoint never gets set up in the first place?
1.) Windows is a closed-source and really huge system. There are many places you will leave traces, and they change all the time. Getting it right is hard.
2.) At least for malware, windows offers official ways to get to go first with e.g. https://learn.microsoft.com/en-us/windows-hardware/drivers/i... - I do not know if this is used by any Anti Cheat though.
The super exotic theory would be a rootkit, in those cases not even windows can help you. But as with security, as long as there is easy money to be made (because most anti-cheat systems are simply bad), those very expensive solutions will be limited to selected few professionals.