Depends if benchmarking is the aim, rather than decreasing the time it takes to build things.
https://huggingface.co/datasets/mbpp?row=98
At the end of the day LLMs in their current iteration aren't intended to do even moderately difficult tasks on their own but it's fun to query them to see progress when new claims are made.
"My favorite thing to ask the models designed for programming is ....... None of them ever get it right"
I read "benchmark".