1) You aren't perfect.
2) Your coworkers aren't perfect.
3) Your "abstractions" are still code that can have errors in it
Also, you are misunderstanding the argument that TDD speeds you up. Yes, at the micro level it may take a bit more time. But, as the system grows, your ability to add or change functionality without breaking things increases when you have tests to verify that it is still correct. (Also, automated testing is much, much faster than manual testing.)