I found it to be my USB mouse. I'd already disabled the option to wake the PC in the mouse device, but then found out that you also have to do that with a separate "keyboard" device[1]. But ever since disabling the option on both devices, things have become a lot better.

[1] https://answers.microsoft.com/en-us/windows/forum/windows_10...