I do spend a lot of time up front specifying what I want. My prompts aren't one-liners, but rather an interview process where we work through all the open questions and ambiguity.
I haven't hit the wall that the OP talked about (when agents just can't seem to make the right changes, and it's impossible for me to go in and change things manually). I used a lot of guardrails such as plan reviews, browser-based QA, adversarial reviews, unit tests, linters, typecheckers, post-commit hooks, and formal method traces. I also specified engineering principles that steers the code base to minimize state and side-effects: functional core; imperative shell, make impossible states impossible, use pure functional style, etc.
There are times, when I can feel a part of the code base is messy without looking at it, because the agent will make recurring mistakes in the same part of the code base over time. What I found the agent was doing over time is that it's been layering state variables as requirements were discovered. So what helps is to ask it to refactor all these state variables into a single sum type. And if the state machine for it is complicated, I'll ask it to write a formal model of the state in Quint. Then I'll generate traces that get run as unit tests, and ask it to write the code against that.
So while the code base isn't exactly Brownfield, it's over a year old now. As for the code base, there's a backend and a frontend. I think it helps that I established a clear pattern I wanted. You code are like memes: agents will just copy patterns they see in the code base. When it does have to create a new part of the system, I found Sonnet-level models tend to draw system boundaries in all the wrong places. Opus is better. I don't yet know about Fable.
Happy to answer any questions about my workflow.