It would also be nice to see a demo where the task was something that I couldn't have done myself in essentially no time. Like, what happens if you say "tasks should support tags, and you should be able to filter/group tasks by tag"?
But if you're curious, task was this:
----
Title: Bug: Users should be able to add tags to a task to categorize them
Description: Users should be able to add multiple tags to a task but aren't currently able to.
Given I am a user with multiple tasks When I select one Then I should be able to add one or many tags to it
Given I am a user with multiple tasks each with multiple tags When I view the list of tasks Then I should be able to see the tags associated with each task
----
And then we ended up with:
GPT-4o ($0.05): https://github.com/sublayerapp/buggy_todo_app/pull/51
Claude 3.5 Sonnet ($0.09): https://github.com/sublayerapp/buggy_todo_app/pull/52
Gemini 2.0 Flash ($0.0018): https://github.com/sublayerapp/buggy_todo_app/pull/53
One thing to note that I've found - I know you had the "...and you should be able to filter/group tasks by tag" on the request - usually when you have a request that is "feature A AND feature B" you get better results when you break it down into smaller pieces and apply them one by one. I'm pretty confident that if I spent time to get the migrations running, we'd be able to build that request out story-by-story as long as we break it out into bite-sized pieces.
If I could just let the AI do all of them and just go in and check the merge requests and approve them it would save me time.