There’s a relevant book from No Starch, highly recommend. The author also has a series of blog posts on the topic.

https://nostarch.com/building-a-debugger