I have three python files (~4k LOC total) that I wanted to refactor with help from Claude 4 (Opus and Sonnet) and I followed Reed Harper's LLM workflow...the results are shockingly bad. It produces an okay plan, albeit full of errors, but usable with heavy editing. In the next step though, most of the code it produced was pretty much unusable. It would've been far quicker for me to just do it myself. I've been trying to get LLMs on various tasks to help me be faster but I'm just not seeing it! There is definitely value in it in helping to straighten out ideas in my head and using it as StackOverflow on roids but that's where the utility starts to hit a wall for me.
Who are these people who are "blown away" by the results and declaring an end to programming as we know it? What are they making? Surely there ought to be more detailed demos of a technology that's purported to be this revolutionary!?
I'm going to write a blog post with what I started with, every prompt I wrote to get a task done and responses from LLMs. Its been challenging to find a detailed writeup of implementing a realistic programming project; all I'm finding is small one off scripts (Simon Willison's blog) and CRUD scaffolding so far.