Very interesting! Good to know that gpt-4 was a step change here.
I tried with 3.5 since that's all I have access to currently, I asked it to do a few things in a ruby app step by step, touching one file at a time, and it worked well! A configuration option that lets you ask the tool to just make file edits, and let me commit later would be useful.
What happens when the files are too large?