I'm using a custom harness based on swe-mini-agent (actually its little brother, their tutorial [0]) and found it way faster than codex (for small tasks) despite being "just bash" in a while loop.
The main difference is that I do the opposite of what you said, i.e. I do dump the context in the prompt. You don't need to grep for what's right in front of you :)
But my repos are small (often smaller than Claude's system prompt!), and I have a script that dumps an "outline" (e.g. function headers and top level vars).
I had an even smaller harness for surgical edits but it was a bit too fiddly and I missed the "let it poke around and figure stuff out" mode of full agentic ones.
What I do miss from that old one though is that it could edit several files simultaneously, in one LLM call. Maybe someday I'll add that back :)
That being said, within a few months everyone who cares about speed will be on Cerebras etc., which will make even the slow harnesses way faster than mine and yours! (I've tested them already and it's insane how fast it is...)