I'd like to see how it handles implementing a Sudoku solver.
I've tried, with multiple different AIs as a benchmark to ask them to implement a Sudoku solver in Rust. And it always turns into an uncompilable mess at worst, or a solver that fills the grid at best but which is not actually following the rules of Sudoku.
Maybe I'm just bad at "prompting". But with all of the material online about implementing your own Sudoku solver I don't understand why any of the models I've asked always produce incorrect solutions. There should be enough info in their training material to know how to do this. I've tried a bunch of different ways like just asking a short simple question without detail ("Implement a Sudoku solver in Rust"), or having it discuss general strategies for the implementation first, along with things like telling it that it's an expert software developer or a professional computer scientist etc. The textual descriptions that it gives sound like they make sense, but when it comes to implementation in code, I've so far never seen actual success with that.
Feeding compile errors from the compiler back into the different LLMs I've tried, often times result in the AI fixing a compile error in one place and then introducing new compile errors in other parts of the code.
If this Artifact thing could enable me to do the following then I am probably going to cancel my OpenAI ChatGPT subscription and start using Claude with Artifacts on the Claude Pro plan instead:
1) Implement a Sudoku solver in Rust. The program will be a command line tool that takes as input a grid of numbers for a Sudoku to solve, and prints the solution to it. It also needs to properly identify invalid input, and input that has no solution. (This alone would be impressive to me because I've never gotten any of the LLMs I've tried to properly do even this.)
2) Make use of serde and hyper to accept a Sudoku puzzle to solve as JSON input over HTTP. Respond with a JSON response that gives the solution to the puzzle.
3) Create a graphical, run-of-the-mill web frontend for the Sudoku solver.
The criteria for success is that the person (me) talking with the LLM never provides any of the actual thoughts about the algorithms and data structures used. If I wanted to do that I'd write it from scratch myself. I want to have the LLM pick the algorithms and data structures, and that the most I do is to paste back any errors and have it actually fix those.