Can AI Code? A self-evaluating junior developer interview for AI models

How often did you get a smartass/extremely literal answer? The example function question has some ambiguity someone who just didn't care could exploit:

    Request: "Write a {{language}} function things() that returns a list with three values: the number 5, the string 'foobar', the capital city of Spain.

[5.0, "fubar", "the capital city of Spain"]

would be a way-too-literal answer, or maybe a test input from an exceptional tester.

Here's a project I've been working on for the last couple of weeks that compares baseline programming performance across multiple LLMs.

ChatGPT is king, which is unsurprising.

What is surprising however is that several smaller models are quite capable: Vicuna-7B 1.1 is pretty decent at Python, while Wizard-Vicuna-13B-Uncensored beats several much larger models at JavaScript (while being the worst at Python).