although these things are impressive at writing hello world in a bunch of ways, I've really found them pretty useless inside any sufficiently complex codebase. Am I just incompetently using them?
Think "here's a schema for some internal API, write me a tool that lets me put in a product ID and see the number of purchases, chargebacks and returns for that product over time, on graphs, also provide sortable by-country tables for these stats".
With artifacts, software goes from something you make once and use for a long time, to something you make at need, to do exactly what you need and no more, and then immediately throw away once it has served its purpose.
Don't ask it how to solve a problem for you, instead write out the solution in plain English, provide an API for it to fill in, and let it write out the solution with unit tests.
This guy [https://x.com/PrajwalTomar_] has been exploring workflows that involve using ChatGPT to assist in creating a Product Requirement Document (PRD), then using V0 by Vercel for mockups and bringing it all together using Cursor, maintaining continuity with markdown documents (.md) of the PRD and relevant database schemas etc inside the project to maintain continuity.
However, I do use copilot+vscode with claude 3.5 and the "Edit with Copilot feature" where it takes my open files, plus any other context I want to give it, to drive changes to my files, has been surprisingly good. It's not really a time saver in that the amount of time I spend verifying and fixing or enhancing the result isn't really faster than me just writing it myself, but I still find benefits for brainstorming, quickly iterating on alternate ideas and smaller refactors, and overcoming the "get started" hesitation on a seemingly complex change. I'm to the point where it can absolutely add tests to files using my existing patterns that are well done and rarely need feedback from me. I have been surprised to the see the progress because for most of the history of LLM I didn't find it useful.
It also helps that I work in a nodejs/react/etc codebase where the models have a ton of information and examples to work with.
There's a checkbox you can toggle so that Openai doesn't use your code to train their models.
And I find the "chatbot" experience different and better than aider/copilot. It forces me to refocus on the really useful interfaces instead of just sending everything, and makes it better to verify everything instead of just accepting a bunch of changes that might even be correct, but not what I exactly want. For me, the time spent verifying is actually a bonus, because I read faster than I can type. I think of it as a peer programmer who just happens to be able to type much, much faster and doesn't mind writing unit tests or rewriting the same thing over and over.
In the edit experience I am using, the LLM provides a git style changelog where I can easily compare before/after with a really detailed diff. I find that much more useful than giant "blobs" of code where minor differences crop up that I don't notice.
The other massive drawback to the out-of-codebase chatbot experience (and the Edit With Copilot experience IS a chatbot, it's just integrated into the editor, changes files with diffs, and has a UI for managing file context) is context. I can effortlessly load all my open files into the LLM context with 1 click. The out-of-editor chatbox requires either a totally custom LLM with various layers to handle your codebase context, or you have to manually paste in a lot of context. It's nonsense to waste time pasting proprietary code into OpenAI (with no business agreement other than a privacy policy and a checkbox) when I can get Copilot to sign a BA with strict rules about privacy and storage, and then 1-click add my open files to my context.
Folks should give these new experiences a try. Having claude chatbot integrated into your editor with the ability to see and modify your open files in a collaborative chat experience is very nice.
I am using windsurf which is similar, they have a two week trial which can give you an idea if it would be useful to you or not.
I use the tab auto-complete and the composer functionality very often.