Moving from a daydream to some actual artefacts the first step _away_ from Idea Debt, not further into it. In technical terms, this could be writing a script instead of building an application or creating a spec and some documentation before writing some code. I'd argue that this is where you start to flesh-out and make real your idea.
Binders of "lore" would be an amazing resource to have at hand when you actually do come to writing your book. Surely this isn't a waste of time but it's where the idea ruminates, develops and matures. If anything, it improves the quality of an idea and can help make that idea a reality.