I'm not familiar with what happens when `debugger` is eval'd while attached using the debugger API. If it ignores it, you're fine.
This method and a few others are described in this paper: https://www.usenix.org/system/files/sec21-musch.pdf
The debugger timing method is described on page 10.