"and timed how long it took them to write an HTTP server in JavaScript"
OK ... interesting choice. This could mean almost anything. I can "write a web server" in a python 1-liner in about 2 seconds (ctrl-r SimpleHttpServer, enter).
OK ... interesting choice. This could mean almost anything. I can "write a web server" in a python 1-liner in about 2 seconds (ctrl-r SimpleHttpServer, enter).
https://github.com/hofstadter-io/demos/tree/main/full-stack-...
I've since shortened that demo to be even faster for a user to complete
the hard bit of being a developer is deciding in precise, pedantic, utterly exact detail on what needs building to solve the business problem
not bashing out the code in the editor
now if they had given out a vague problem that ended up with the dev eventually producing a webserver in Javascript, then fine
but then the difference copilot makes would be a much smaller percentage of the development effort
I swear we're going back to the age of "lines of code written per hour", just now with arTiFiCiAL InTellIgEncE
The process seemed to involve have a test suite to write against. So, essentially, you had some/most of your "business problem" defined by acceptance tests (produce these outputs given these inputs). Some of that process may normally stretch out for days or weeks during meetings - condensed down, provided for you ahead of time takes some of that variability out.
I was possibly a bit more surprised that less than 80% finished. Did they not commit code? Did they submit code but it didn't pass? Did they take too long? Was the 'did not finish' an explicit call on the dev side ("I'm done") or did they just stop taking input after X days?