I used this to scaffold out 5 HTML pages for a web app, having it iterate on building the UX. Did a pretty good job and took about 10 minutes of iterating with it, but cost me about $10 in API credits which was more than I expected.
Cost is one of our biggest issues right now. There's a lot we can do to mitigate, but we've been focused on getting something that works well before optimizing for efficiency.
I think that’s correct – even at a “high” cost (relative to what? A random SaaS app or an hour of a moderately competent Full Stack Dev?) the ROI will already be there for some projects, and as prices naturally improve a larger and larger portion of projects will make sense while we also build economies of scale with inference infrastructure.
This is a bigger issue than folks realize, visual inputs to GPT4 are really expensive (like several cents per dozen images in some cases), which means that you can't just spam the API to iterate on HTML/webpages with a software agent. We're trying to tackle this for web screenshots (also documents) with a custom model geared towards structured schemas designed to be fed into a feedback loop like the above while keeping costs down.