Simon Willison on Phoenix.new

I decided to give Phoenix.new a spin to see where it stood in the field. Overall? Cool DevX but I'm not super impressed with the actual output.

I dumped $110 into it over the course of a day, just to give it a fair shake and see how far I could bring it. The first $40 or so was pretty productive and I had a cool little prototype. The last $70, however, was basically stuck in a death loop where it couldn't fix certain (obvious) problems.

It would read my feedback, do some tests (this feature is super cool by the way), and then declare that everything works. I would then click through the UI, see that nothing changed, and then reprompt. It would declare that, indeed, there is an error! It would go about on a tangent trying to solve it, declare victory, and then give a cutsey update about how it solved the issue. I then test, see that nothing changed, and then reprompt... ad infinitum (or at least until I stopped putting tokens in the machine). Honestly, now that I think about it, it feels like I paid these guys to get the standard Product Manager experience.

Of course, I can't be too harsh. This has been pretty much my experience with most AI products. I was hoping the way Phoenix.new was set up (where it plans it's work and tests itself) would achieve better results, but that seems not to really make a difference.

Actual title: "Phoenix.new is Fly’s entry into the prompt-driven app development space"