Granted that particular team was an extremely low-skill team in general, and I've used TDD now and again successfully when it fit my needs, but that experience was a pretty strong counter-example to the strong claims made by TDD proponents about how it improves the quality of code.
We write almost entirely integration tests (and lots of them) and we rarely need to maintain tests since they're only tied to the UI. Most refactors happen w/ no test changes at all. Occasionally a UI change causes a big find/replace across lots of tests.
The tests are also probably less comprehensive than a suite of finer grained tests would be. Most tests tend to cover happy path and few edge cases.
When we started there were zero tests and refactoring was too risky so the early code decisions largely had to follow the architecture that was already present (which wasn't always what we wanted). As we filled out the tests for existing behavior we could refactor more.
I think the mostly-integration-testing approach led us to a pretty good balance by only coupling the test to the UI (with its own cons of course).
This way I'm not rewriting tests every time I need to rip apart and restructure the internals but I still have the safety net of refactoring without breaking the external contract.