I generally find that it's better to indulge my lack of focus in a controlled manner than it is to try fighting it.
I can wake up in the middle of the night with a great idea to improve a UI workflow, or to squeeze more performance out of a process, but if my disciple doesn't kick and I write the code to get it done,
Also it doesn't take intelligence to makes sure my Press page, About page and Resume stay current, it just takes discipline.
All of my prior jobs have been given to me based on my completed portfolio projects, not on my great ideas. And numerous co-workers have been let go because their brilliance never results in 'finished' projects.
While I can see bursts of undirected creativity stimulating small, quick discoveries, anything that takes longer than, say, a week (probably more like a day), just doesn't seem achievable.
Then again, you said your "best work", not biggest. I suppose those are actually be mutually exclusive in many cases.
The stuff that can be finished in a week often feels like my best work, though, at least intellectually (and you can get a good amount done in a week if you're excited about it). The week-long side projects tend to have more novelty, which sometimes even results in more impactful published papers: you're proposing the first X to do Y, and you have a working system to boot (in academia not always required, but a nice plus if you have it).
My biggest projects tend to require some focus and months, but they tend to be more of the grind variety. Take that initial prototype, either one I've produced or one someone else has published, and build a Real System on it, working out all the details, including engineering and theory as appropriate.
A small, quick project that does just one thing well is probably a lot easier to see as your "best" because there's just fewer edge cases for you to say "boy, I wish I would have done it this other way..."
Generally speaking, my approach to programming is more trial and error than organized and methodical. But then again, my work style doesn't work in all situations nor does it work for everyone else. I'm generally better at starting a project than finishing one. However, just like some projects take longer to finish than others, some projects take longer to start than others. So I I don't think that time is terribly pertinent.
With that in mind, a certain amount of distractibility is a blessing for me. If something distracts me, that means it will distract whoever is finishing a project. Therefore, it's important to address those distractions ahead of time.