Each process can be ptraced by only once process at any time, so if the malicious program want to avoid debuggers, they can attempt a ptrace() on their own process. If it fails, they know they are being debugged.