Retrowin32: Async, DLL loading, tracing execution, and Zig
neugierig.org
neugierig.org
Go deep enough into understanding Windows architecture and you'll find that there's a lot of officially-undocumented stuff that various "protections" rely on. PEB/TEB fields, syscall entry points, known fixed addresses, anti-debugging/VM detection, etc.
As for that particular scary message, I recognised it as coming from a packer I saw a long time ago --- and a quick search reveals it is PEtite:
http://users.freenet.am/~softland/tutorials/Petite.v2.3.MUP....
Packed software is in general a good way to "stress test" your emulator. Besides the obvious fact of being essentially self-modifying code, they also tend to exercise the more subtle aspects of the OS loader/linker.
Are such syscalls just defined in the winapi and abstracted away?
Yes. Unlike Linux, the OS interfaces with the applications via system DLLs (kernel32, user32, gdi32, etc.)
As a historical note, the syscall interface on Win9x is completely different from the NT series.
Which I realize is brilliant for many people, just not me, don't have the necessary attention/time-tokens available for yet another it hobby sub!
Once upon a time(tm), I took advantage of the DLL directory search order to crack or change behavior in annoying software. I would compile a proxy DLL to a system DLL that would handle the symbols I was interested in and send all other calls onto the real DLL unmodified. I forgot the mechanism and it probably doesn't work anymore in Windows 11.
Edit: Basically:
C:\Program Files\Brand\Some App\SYSTEM32.DLL <- mine and this one is loaded ahead of the real system one.