Artifacts are now generally available
anthropic.com
anthropic.com
For he sake of it, to test the waters would you say, I decided to prompt for a small React app I've had in my mind [1] (Link below app description) for a couple of days now. I know this is far from being a polished, market ready app. But I do think Junior Devs are all outta job (Or going to be, within 6 - 18 months). Prepare for the shift, this one particular paradigm is taking no prisoners.
---
For the record, this is the app I got out of it, fully working:
My GF owns a cafeteria and she's running a promo on all her menu, _except_ other promos. The promotion is a 20% discount on all products, with the price rounded to the nearest 100 pesos (Argentina).
The small app I requested is single page, with a large number (total price) displayed on top. Below is a 3 tabbed panel, with tabs being PROMO, REGULAR and SELECTED Each tab has a list of products, with buttons [ - ][ # ] [ + ] <name> $<price> When a [ + ] button is clicked, the number increases, and that product appears in the SELECTED tab if it wasn't there. [ - ] decreases the # in both listings, and if its zero it disappears from selected. The price on top is always updated to reflect the total price. The SELECT tab has all the selected products (You can change the # for each product there as well) with price, and a per type subtotal (as the discount is only applied to REGULAR products). The subtotal when discounted appears as ~$#.##~ $#.## where the first instance of a number is the price without discount and the 2nd number is the price after discount.
My knowledge of React is negligible, and even if I know it enough to do this it would had taken me at least a couple hours to code. I could probably deploy it as an app to my GF's phone in less than an hour compiling and all.
It has also the possibility of an API call, so I could shove all them products in a page and go changing accordingly.
[1] https://claude.site/artifacts/2fe3c97d-88f6-4e2f-9ed2-96452e...
I'd like to see how it handles implementing a Sudoku solver.
I've tried, with multiple different AIs as a benchmark to ask them to implement a Sudoku solver in Rust. And it always turns into an uncompilable mess at worst, or a solver that fills the grid at best but which is not actually following the rules of Sudoku.
Maybe I'm just bad at "prompting". But with all of the material online about implementing your own Sudoku solver I don't understand why any of the models I've asked always produce incorrect solutions. There should be enough info in their training material to know how to do this. I've tried a bunch of different ways like just asking a short simple question without detail ("Implement a Sudoku solver in Rust"), or having it discuss general strategies for the implementation first, along with things like telling it that it's an expert software developer or a professional computer scientist etc. The textual descriptions that it gives sound like they make sense, but when it comes to implementation in code, I've so far never seen actual success with that.
Feeding compile errors from the compiler back into the different LLMs I've tried, often times result in the AI fixing a compile error in one place and then introducing new compile errors in other parts of the code.
If this Artifact thing could enable me to do the following then I am probably going to cancel my OpenAI ChatGPT subscription and start using Claude with Artifacts on the Claude Pro plan instead:
1) Implement a Sudoku solver in Rust. The program will be a command line tool that takes as input a grid of numbers for a Sudoku to solve, and prints the solution to it. It also needs to properly identify invalid input, and input that has no solution. (This alone would be impressive to me because I've never gotten any of the LLMs I've tried to properly do even this.)
2) Make use of serde and hyper to accept a Sudoku puzzle to solve as JSON input over HTTP. Respond with a JSON response that gives the solution to the puzzle.
3) Create a graphical, run-of-the-mill web frontend for the Sudoku solver.
The criteria for success is that the person (me) talking with the LLM never provides any of the actual thoughts about the algorithms and data structures used. If I wanted to do that I'd write it from scratch myself. I want to have the LLM pick the algorithms and data structures, and that the most I do is to paste back any errors and have it actually fix those.
Edit: tried with Llama 3.1 70B as well, and funnily enough it outputs the exact same Sudoku puzzle as an example to test the code with.
The app description sounds like it was more work to come up with than the app code itself.
Also, this is an extremely common, bland and simplistic example. You’ll see it flail and falter on slightly less generic boilerplate-heavy “apps” and there is no known tech/theory that’ll get us there. We just hope more “scale” will magically solve all its issues, but IMO that’s not going to happen. The paradigm just doesn’t allow it. Time will tell who is right of course.
I don’t want to downplay the recent LLM achievements because they are amazing, but this is not taking any (programming) jobs.
I’m actually not too worried about the current wave of AI. Copilot is an annoyance and GPT4 is lying so much it’s getting old. LLMs have deep issues that I’m not seeing much improvement on. The next wave of AI however.. I’m not sure. It’s a matter of time.
Claude appears to be really good at Javascript-based conversations.
Also, a search functionality which searches in existing conversation would be at the top of my wish list. AFAIK only Mistral offers this. Gemini is the worst at giving you access to older conversations, where you have to use the "Activity" functionality.
I found that for anything above hello world complexity the snippets produced by these tools take more time and have lower quality than if you just learned the tool yourself (which you can use AI for) and did it yourself.
I did manage to get interesting results when I turned it into a few shot prompt with a lot more RAG. So maybe more specialized versions of these tools can produce better results than generalist versions.
The advantage is this lets you see what something would look like in a few minutes without needing to learn the thing (saves you 2.5 hours) the disadvantage is when you try to add new features or build a new site you haven’t learned anything. The 2.5 hours is an investment in the skill.
Another interesting thing to point out is how the UI pattern is off in your example. Do you see how “selected” category is in the same multi selection as the actual menu items whereas the normal UI pattern would be to have a Cart or something on the page. This is the issue that I have been running into with these Generative UI approaches is this stuff happens constantly and in very strange ways.
I don't have an example to prove you wrong here, but here's my 2 cents:
1. You could probably decompose the app in different parts, and then put everything together. Even thought my experience with React is nill, I do (Think I do) have the expertise and know-how to glue everything together (It is yet to be proven that Claude cannot do this). 2. I obviously don't think it can replace a Sr. Dev (yet) but for somebody with my expertise, this is a very good starting point. It saves you 2.5 hours... Every. Single. Time. 3. This is ground zero (sort of). In the words of the famous Tank from The Matrix... This is loco.
The good part is you can get the minimum viable product of every feature up every time. The bad part is when you run into bugs you can’t debug it yourself because you haven’t built up the mental model of how the system works since you’re just copying and pasting. The more unique your problem gets (once you’re out the hello world zone of problems) the worse the AI is at helping you. The first few times you run into a simple debugging and the AI figures it out for you it’s pretty cool but the frequent times the AI gets stuck in a loop and you spend hours feeding it debug output after debug output and it still being broken makes you hate this workflow.
So yes it does help you rapidly build the features but at the same time there’s an immense amount of additional work needed to maintain and debug the system. Also the code is generally very poorly structured and architected.
So yea impressive stuff at a surface level and great for speeding up your ability to get started on a task. But really bad to rely on it (at this point) for anything of greater complexity than the basic quick start tutorial of your library of choice (in my experience).
Sounds interesting. How does this work? Is it any good?
I’ve been enjoying Claude, but while the output is single file instead of a codebase, it’s pretty limited and toy-ish.
> Both Claude's projects and context size make GPT a child's toy (IMHO)
I've found GPT still seems to give me better responses for some types of things, but its hard to quantify. I may just be hallucinating the quality difference.