I'll add "having to deal with broken this more often than working things".

Building a feature? You're basically fighting compilation errors, editor warnings, spelling mistakes, your own incompetence, etc. And once it's working, you stop looking at it: it's shipped, you're off to the next task!

Fixing a bug? You now have to look for something that's broken or, even worse, make it break on purpose. Then you're basically back to the "building a feature" workflow, and off to the next task!

This is what I find the most frustrating, to be honest!