Designing tests at the unit level requires you to design low-level/"tactical" aspects of your software, which automatically forces you to commit to a certain general/"strategic" design of the overall program. This is often the opposite of what you want in greenfield work - you want to first rapidly explore the space of high-level designs, not precommit to the first one you thought of that makes tests easier.

Starting with TDD is like being a general whose recommendation to every international politics issue is to commence training for a combined arms assault on the capital of the country's historical enemy, because that's something that can be iterated to provide good estimates and error-free execution. Except, that general misses other, possibly much more effective approaches, such as airstrikes, assassinations, supplying domestic terrorists in enemy territory, or... not shooting anything at all because we're not at war and the problem can be trivially solved by respective VPs over some drinks. Or the problem isn't even a problem.

Keep the TDD General around for when you're actually fighting a land war - but for the love of $deity, don't let him drive international policy in peace time.