Chat is a bad UI pattern for development tools
danieldelaney.net
danieldelaney.net
I just finished a small project where I used o3-mini and o3-mini-high to generate most of the code. I averaged around 200 lines of code an hour, including the business logic and unit tests. Total was around 2200 lines. So, not a big project, but not a throw away script. The code was perfectly fine for what we needed. This is the third time I've done this, and each time I get faster and better at it.
1. I find a "pair programming" mentality is key. I focus on the high-level code, and let the model focus on the lower level code. I code review all the code, and provide feedback. Blindly accepting the code is a terrible approach.
2. Generating unit tests is critical. After I like the gist of some code, I ask for some smoke tests. Again, peer review the code and adjust as needed.
3. Be liberal with starting a new chat: the models can get easily confused with longer context windows. If you start to see things go sideways, start over.
4. Give it code examples. Don't prompt with English only.
FWIW, o3-mini was the best model I've seen so far; Sonnet 3.5 New is a close second.
Works amazingly well for a lot of what I've been working on the past month or two.
Copilot, back when I used it, completely ignored context outside of the file I was working in. Copilot, as of a few weeks ago, the absolute dumbest assistant of all the various options available.
With cursor, I can ask it to make a change to how the app generates a JWT without even knowing which file or folder the relevant code is in. For very large codebases, this is very very helpful.
The minute you have to discuss those things with someone else, your bandwidth decreases by orders of magnitude and now you have to put words to these things and describe them, and physically type them in or vocalize them. Then your counterpart has to input them through his eyes and ears, process that, and re-output his thoughts to you. Slow, slow, slow, and prone to error and specificity problems as you translate technical concepts to English and back.
Chat as a UX interface is similarly slow and poorly specific. It has all the shortcomings of discussing your idea with a human and really no upside besides the dictionary-like recall.
Especially with the new r1 thinking output, I find it useful to iterate on the initial prompt as a way to make my ideas more concrete as much as iterating through the chat interface which is more hit and miss due to context length limits.
* I don’t mean that in a negative way, but in a “I can’t expect another person to respond to me instantly at 10 words per second” way.
I mean, isn’t typing your code also forcing you to make your ideas concrete
I’m not saying pair programming is a silver bullet, and I tend to agree that working on your own can be vastly more efficient. I do however think that it’s a very useful tool for critical functionality and hard problems and shouldn’t be dismissed.
I've been coding long enough to notice there are times where the problem is complex and unclear enough that my own thought process will turn into pair programming with myself, literally chatting with myself in a text file; this process has the bandwidth and latency on the same order as talking to another person, so I might just as well do that and get the benefit of an independent perspective.
The above is really more of a design-level discussion. However, there are other times - precisely those times that pair programming is meant for - when the problem is clear enough I can immerse myself in it. Using the slow I/O mode, being deliberate is exactly the opposite of what I need then. By moving alone and focused, keeping my thoughts below the level of words, I can explore the problem space much further, rapidly proposing a solution, feeling it out, proposing another, comparing, deciding on a direction, noticing edge cases and bad design up front and dealing with them, all in a rapid feedback loop with test. Pair programming in this scenario would truly force me to "use the slower I/O parts of your brain", in that exact sense: it's like splitting a highly-optimized in-memory data processing pipeline in two, and making the halves communicate over IPC. With JSON.
As for bus factor, I find the argument bogus anyway. For that to work, pair programming would've to be executed with the same partner or small group of partners, preferably working on the same or related code modules, daily, over the course of weeks at least - otherwise neither them nor I are going to have enough exposure to understand what the other is working on. But it's not how pair programming worked when I've experienced it.
It's a problem with code reviews, too: if your project has depth[0], I won't really understand the whole context of what you're doing, and you won't understand the context of my work, so our reviews of each others' code will quickly degenerate to spotting typos, style violations, and peculiar design choices; neither of us will have time or mental capacity to fully understand the changeset before "+2 LGTM"-ing it away.
--
[0] - I don't know if there's a a better, established term for it. What I mean is depth vs. breadth in the project architecture. Example of depth: you have a main execution orchestrator, you have an external data system that handles integrations with a dozen different data storage systems, then you have math-heavy business logic on data, then you have RPC for integrating with GUI software developed by another team, then you have extensive configuration system, etc. - each of those areas is full of design and coding challenges that don't transfer to any other. Contrast that with an example of breadth: a typical webapp or mobile app, where 80% of the code is just some UI components and a hundred different screens, with very little unique or domain-specific logic. In those projects, developers are like free electrons in metal: they can pick any part of the project at any given moment and be equally productive working on it, because every part is basically the same as every other part. In those projects, I can see both pair programming and code reviews deliver on their promises in full.
Peer and near-peer reviews have always wound up being nitpicking or perfunctory.
An alternative that might work if you want two hands on every change for process reasons is to have the reviewer do something closer to formal QA, building and running the changed code to verify it has the expected behavior. That has a lot of limitations too, but it least it doesn’t degrade to bikeshedding about variable name aesthetics.
For me pair programming accelerates development to much more than 2x. Over time the two of you figure out how to use each other's strengths, and as both of you immerse yourself in the same context you begin to understand what's needed without speaking every bit of syntax between each other.
In best cases as a driver you end up producing high quality on the first pass, because you know that your partner will immediately catch anything that doesn't look right. You also go fast because you can sometimes skim over complexities letting your partner think ahead and share that context load.
I'll leave readers to find all the caveats here
Edit: I should probably mention why I think Chat Interface for AI is not working like Pair programming: As much as it may fake it, AI isn't learning anything while you're chatting to it. Its pointless to argue your case or discuss architectural approaches. An approach that yields better results with Chat AI is to just edit/expand your original prompt. It also feels less like a waste of time.
With Pair programming, you may chat upfront, but you won't reach that shared understanding until you start trying to implement something. For now Chat AI has no shared understanding, just "what I asked you to do" thing, and that's not good enough.
For me, there's
- Time when I want to discuss the approach and/or code to something (someone being there is a requirement)
- Time when I want to rubber duck, and put things to words (someone being there doesn't hurt, but it doesn't help)
- Time when I want to write code that implements things, which may be based on the output of one of the above
That last bucket of time is generally greatly hampered by having someone else there and needing to interact with them. Being able to separate them (having people there for the first one or two, but not the third) is, for me, optimal.
Maybe collaborate the first hour each morning, then the first hour after lunch.
What's your definition of 'learn'? An LLM absolutely does extract and store information from its context. Sure, it's only short term memory and it's gone the next session, but within the session it's still learning.
I like your suggestion to update your original prompt instead of continuing the conversation.
with LLMs that leaning is only context, and in case of “chat ai” is the chat backlog. It’s easy for the LLM to get stuck, and there aren’t many tools yet that help you change and morph that context to be more like shared understanding between two colleagues.
There is research going on in this area, eg “chain of thought”, so we’ll see, maybe things will get better
The value of pair programming is inversely proportional to the expertise of the participant. Junior devs who pair with senior devs get a lot out of it, senior devs not so much.
GP is probably a more experienced dev, whereas you are the type of dev who says things like “I’m guessing that you…”.
In pair programming as I learned it and as I have occasionally experienced it, two individuals challenge each other to be their best selves while also handing off tasks that break flow so that the pair as a whole is in constant flow. When it works this is a fantastic, productive, intense experience. I would agree that it is more than 2x as productive. I don't believe it's possible to achieve this state at all with a mismatched pair.
If this is the experience people have in mind, then it's not surprising that they think that those who think it's only for training juniors haven't actually tried it very much.
One interesting thing is skill levels aren’t really comparable or equatable. I find pairing is productive where skills only partially overlap, meaning the authority on parts of implementation flows between participants, depending on the area you’re in.
I have some examples where I recently paired with a colleague for about 2-3 weeks nearly every day. i’m confident that what took us a month would be near impossible just working on our own
That's very much not my experience. Pairing on design and diagrams is as or more useful than on the code itself. Once you have a good design, the code is pretty simple.
I wrote about this here: https://digitalseams.com/blog/the-ideal-ai-interface-is-prob...
Neural networks and evolved structures and pathways (e.g. humans make do with ~20k genes and about that many more in regulatory sequences) are absolutely more efficient, but good luck debugging them.
Pair programming is endlessly frustrating beyond just rubber duckying because I’m having to exit my mental model, communicate it to someone else, and then translate and relate their inputs back into my mental model which is not exactly rooted in language in my head.
Subvocalization/explicit vocalization of what you're doing actually improves your understanding of the code. Doing so may 'decrease bandwith', but improves comprehension, because it's basically inline rubber duck debugging.
It's actually easy to write code which you don't understand and cannot explain what it's doing, whether at the syntax, logic or application level. I think the analogue is to writing well; anyone can write streams of consciousness amounting to word salad garbage. But a good writer can cut things down and explain why every single thing was chosen, right down to the punctuations. This feature of writing should be even more apparent with code.
I've coded tons of things where I can get the code working in a mediocre fashion, and yet find great difficulty in try to verbally explain what I'm doing.
In contrast there's been code where I've been able to explain each step of what I'm doing before I even write anything; in those situations what generally comes out tends to be superior maintainable code, and readable too.
our brains like to jump over inconsistencies or small gaps in our logic when working by themselves, but try to explain that same concept to someone else and those inconsistencies and gaps become glaringly obvious (doubly so if the other person starts asking questions you never considered)
it's why pair programming and rubber duck debugging work at all, at least in my opinion
it would seem to me that would cause a lot of issues
Yesterday, I asked o3-mini to "optimize" a block of code. It produced very clean, functional TypeScript. However, because the code is reducing stock option chains, I then asked o3-mini to "optimize for speed." In the JavaScript world, this is usually done with for loops, and it even considered aspects like array memory allocation.
This shows that using the right qualifiers is important for getting the results you want. Today, I use both "optimize for developer experience" and "optimize for speed" when they are appropriate.
Although declarative code is just an abstraction, moving from imperative jQuery to declarative React was a major change in my coding experience. My work went from telling the system how to do something to simply telling it what to do. Of course, in React—especially at first—I had to explain how to do things, but only once to create a component. After that, I could just tell the system what to do. Now, I can simply declare the desired outcome, the what. It helps to understand how things work, but that level of detail is becoming less necessary.
I paste that into Claude and it is surprisingly good at fixing bugs and making visual modifications.
This is what I've found to be key. If I start a new feature, I will work with the LLM to do the following:
- Create problem and solution statement
- Create requirements and user stories
- Create architecture
- Create skeleton code. This is critical since it lets me understand what it wants to do.
- Generate a summary of the skeleton code
Once I have done the above, I will have the LLM generate a reusable prompt that I can use to start LLM conversations with. Below is an example of how I turn everything into a reusable prompt.
https://beta.gitsense.com/?chat=b96ce9e0-da19-45e8-bfec-a3ec...
As I make changes like add new files, I will need to generate a new prompt but it is worth the effort. And you can see it in action here.
https://beta.gitsense.com/?chat=b8c4b221-55e5-4ed6-860e-12f0...
The first message is the reusable prompt message. With the first message in place, I can describe the problem or requirements and ask the LLM what files it will need to better understand how to implement things.
What I am currently doing highlights how I think LLM is a game changer. VCs are going for moonshots instead of home runs. The ability to gather requirements and talk through a solution before even coding is how I think LLMs will revolutionize things. It is great that it can produce usable code, but what I've found it to be invaluable is it helps you organize your thoughts.
In the last link, I am having a conversation with both DeepSeek v3 and Sonnet 3.5 and the LLMs legitimately saved me hours in work, without even writing a single line of code. In the past, I would have just implemented the feature and been done with it, and then I would have to fix something if I didn't think of an edge case. With LLMs, it literally takes minutes to develop a plan that is extremely well documented that can be shared with others.
This ability to generate design documents is how I think LLMs will ultimately be used. The bonus is producing code, but the reality is that documentation (which can be tedious and frustrating) is a requirement for software development. In my opinion, this is where LLMs will forever change things.
As a side note, "No, you're wrong" is not a great way to have a conversation.
Sometimes typos are eerily appropriate ;)
(I almost typed "errily"...)
Came to vote good too. I mean, why do we all love a nice REPL? That's chat right? Chat with an interpreter.
1. I need a smart autocomplete that can work backwards and mimic my coding patterns
2. I need a pair programming buddy (of sorts, this metaphor doesn't completely work, but I don't have a better one)
Pair development, even a butchered version of the so called "strong style" (give the driver the highest level of abstraction they can use/understand) works quite well for me. But, the main reason this works is that it forces me to structure my thinking a little bit, allows me to iterate on the definition of the problem. Toss away the sketch with bigger parts of the problem, start again.
It also helps me to avoid yak shaving, getting lost in the detail or distracted because the feedback loop between me seeing something working on the screen vs. the idea is so short (even if the code is crap).
I'd also add 5.: use prompts to generate (boring) prompts. For instance, I needed a simple #tag formatter for one of my markdown sites. I am aware that there's a not-so-small list of edge cases I'd need to cover. In this case I'd write a prompt with a list of basic requirements and ask the LLM to: a) extend it with good practice, common edge cases b) format it as a spec with concrete input / output examples. This works a bit similar to the point you made about generating unit tests (I do that too, in tandem with this approach).
In a sense 1) is autocomplete 2) is a scaffolding tool.
- Text prompts and chat interfaces are great for coarse grained exploration. You can get a rough start that you can refine. "Knight standing in a desert, rusted suit of armor" gets you started, but you'll want to take it much further.
- Precision inputs (mouse or structure guided) are best for fine tuning the result and honing in on the solution itself. You can individually plant the cacti and pose the character. You can't get there with text.
> 1. I need a smart autocomplete that can work backwards and mimic my coding patterns
> 2. I need a pair programming buddy (of sorts, this metaphor doesn't completely work, but I don't have a better one)
Thanks! This is the first time I've seen it put this clearly. When I first tried out CoPilot, I was unsure of how I was "supposed" to interact with it. Is it (as you put it) a smarter autocomplete, or a programming buddy? Is it both? What was the right input method to use?
After a while, I realized that for my personal style I would pretty much entirely use method 1, and never method 2. But, others might really need that "programming buddy" and use that interface instead.
Example: - copy paste a table from a pdf datasheet into a comment (it'll be badly formatted with newlines and whatnot, doesn't matter) - show it how to do the first line - autocomplete the rest of the table - Check every row to make sure it didn't invent fields/types
For this type of workflow the tools are a real time saver. I've yet to see any results for the other workflows. They usually just frustrate me by either starting to suggest nonsense code without full understanding, or its far too easy to bias the results and make them stuck in a pattern of thinking.
Chat sucks for pulling in context, and the only worse thing I've tried is the IDE integrations that supposedly pull the relevant context for you (and I've tried quite a few recently).
I don't know if naive fine-tuning with codebase would work, I suspect there are going to be tools that let you train the AI on code in the sense that it can have some references in model, and it knows how you want your project code/structure to look like (which is often quite different from what it looks in most areas)
> I focus on the high-level code, and let the model focus on the lower level code.
I've yet to see any model understand nuance or detail.
This is especially apparent in image models. Sure, it can do hands but they still don't get 3D space nor temporal movements. It's great for scrolling through Twitter but the longer you look the more surreal they get. This even includes the new ByteDance model also on the front page. But with coding models they ignore context of the codebase and the results feel more like patchwork. They feel like what you'd be annoyed at with a junior dev for writing because not only do you have to go through 10 PRs to make it pass the test cases but the lack of context just builds a lot of tech debt. How they'll build unit tests that technically work but don't capture the actual issues and usually can be highly condensed while having greater coverage. It feels very gluey, like copy pasting from stack overflow when hyper focused on the immediate outcome instead of understanding the goal. It is too "solution" oriented, not understanding the underlying heuristics and is more frustrating than dealing with the human equivalent who says something "works" as evidenced by the output. This is like trying to say a math proof is correct by looking at just the last line.
Ironically, I think in part this is why chat interface sucks too. A lot of our job is to do a lot of inference in figuring out what our managers are even asking us to make. And you can't even know the answer until you're part way in.
Have you tried Cursor? It has a great feature that grabs context from the codebase, I use it all the time.
I actually cancelled by Anthropic subscription when I started using cursor because I only ever used Claude for code generation anyway so now I just do it within the IDE.
Here's a simple example with GPT-4o: https://0x0.st/8K3z.png
It probably isn't obvious in a quick read, but there are mistakes here. Maybe the most obvious is that how `replacements` is made we need to intelligently order. This could be fixed by sorting. But is this the right data structure? Not to mention that the algorithm itself is quite... odd
To give a more complicated example I passed the same prompt from this famous code golf problem[0]. Here's the results, I'll save you the time, the output is wrong https://0x0.st/8K3M.txt (note, I started command likes with "$" and added some notes for you)
Just for the heck of it, here's the same thing but with o1-preview
Initial problem: https://0x0.st/8K3t.txt
Codegolf one: https://0x0.st/8K3y.txt
As you can see, o1 is a bit better on the initial problem but still fails at the code golf one. It really isn't beating the baseline naive solution. It does 170 MiB/s compared to 160 MiB/s (baseline with -O3). This is something I'd hope it could do really well on given that this problem is rather famous and so many occurrences of it should show up. There's tons of variations out there and It is common to see parallel fizzbuzz in a class on parallelization as well as it can teach important concepts like keeping the output in the right order.
But hey, at least o1 has the correct output... It's just that that's not all that matters.
I stand by this: evaluating code based on output alone is akin to evaluating a mathematical proof based on the result. And I hope these examples make the point why that matters, why checking output is insufficient.
[0] https://codegolf.stackexchange.com/questions/215216/high-thr...
Edit: I want to add that there's also an important factor here. The LLM might get you a "result" faster, but you are much more likely to miss the learning process that comes with struggling. Because that makes you much faster (and more flexible) not just next time but in many situations where even a subset is similar. Which yeah, totally fine to glue shit together when you don't care and just need something, but there's a lot of missed value if you need to revisit any of that. I do have concerns that people will be plateaued at junior levels. I hope it doesn't cause seniors to revert to juniors, which I've seen happen without LLMs. If you stop working on these types of problems, you lose the skills. There's already an issue where we rush to get output and it has clear effects on the stagnation of devs. We have far more programmers than ever but I'm not confident we have a significant number more wizards (the percentage of wizards is decreasing). There's fewer people writing programs just for fun. But "for fun" is one of our greatest learning tools as humans. Play is a common trait you see in animals and it exists for a reason.
If only this feature worked consistently, or reliably even half of the time.
It will casually forget or ignore any and all context and any and all files in your codebase at random times, and you never know what set of files and docs it's working with at any point in time
That's interesting. I found assistants like Copilot fairly good at low level code, assuming you direct it well.
> assuming you direct it well.
I do think that when I'm coding with an LLM it _feels_ faster, but when I've timed myself, it doesn't seem that way. It just seems to be less effort (I don't mind the effort, especially because the compounding rewards).
This is why I think LLMs can't really replace developers. 80% of my job is already trying to figure out what's actually needed, despite being given lots of text detail, maybe even spec, or prototype code.
Building the wrong thing fast is about as useful as not building anything at all. (And before someone says "at least you now know what not to do"? For any problem there are infinite number of wrong solutions, but only a handful of ones that yield success, why waste time trying all the wrong ones?)
Devil's advocate: because unless you're working in heavily dysfunctional organization, or are doing a live coding interview, you're not playing "guess the password" with your management. Most of the time, they have even less of a clue about how the right solution looks like! "Building the wrong thing" lets them diff something concrete against what they imagined and felt like it would be, forcing them to clarify their expectations and give you more accurate directions (which, being a diff against a concrete things, are less likely to be then misunderstood by you!). And, the faster you can build that wrong thing, the less money and time is burned to buy that extra clarity.
Doing 4 sprints over 2 months to make a prototype in order to save 3 60 minute meetings over a week where you do a few requirements analysis/proposal review cycles.
I see this not as opposed, but as part of requirements analysis/review - working in the abstract, with imagination and prose and diagrams, it's too easy to make invalid assumptions without anyone realizing it.
That’s a lot of effort for a prototype that you should be throwing away even if it does the right thing!
Are you sure you’re not gold plating your prototypes?
> you can't even know the answer until you're part way in.
It is really hard to figure out what the right thing is. We humans don't do this just through chat. We experiment, discuss, argue, draw, and there's tons of inference and reliance upon shared understandings. There's a lot of associated context. You're right that a dysfunctional organization (not uncommon) is worse, but these things are still quite common in highly functioning organizations. Your explicit statement about management having even less of an idea of what the right solution is, is explicitly what we're pushing back against. Saying that that is a large part of a developer's job. I would argue that the main reason we have a better idea is due to our technical skills, our depth of knowledge, our experience. A compression machine (LLM) will get some of this, but there's a big gap when trying to get to the end point. Pareto is a bitch. We all know there is a huge difference between a demonstrating prototype and an actual product. That the amount of effort and resources are exponentially different. ML systems specifically struggle with detail and nuance, but this is the root of those resource differences.
I'll give an example for clarity. Considering the iPad, the existence of third party note taking apps can be interpreted of nothing short of Apple's failure. I mean for the love of god, you got the pencil and you can't pull up notes and interact with it like it is a piece of paper? It's how the damned thing is advertised! A third party note taking app should be interpreted by Apple as showing their weak points. But you can't even zoom on the notes app?! Sure, you can turn on the accessibility setting and zoom with triple tap (significantly diverging from the standard pinching gesture used literally everywhere else) but if you do this (assuming full screen) you are just zooming in on the portion of the actual screen and not zooming in the notes. So you get stupid results like not having access to your pen's settings. Which is extra important here given that the likely reason someone would zoom is to adjust details and certainly you're going to want to adjust the eraser size. What I'm trying to say is that there's a lot of low hanging fruit here that should be incredibly obvious were you to actually use the application, dog-fooding. Instead, Apple is dedicating time into hand writing recognition and equation solving, which in practice (at least in my experience) end up creating a more jarring experience and cause more editing. Though it is cool when it does work. I'd say that here, Apple is not building the right thing. They are completely out of touch with the actual goals and experiences of the users. It's not that they didn't build a perfect app, it is that they fail to build basic functionality.
But of course, Apple probably doesn't care. Because they significantly prioritize profits over building a quality product. These are orthogonal aspects and they can be simultaneously optimized. One should not need pick one over another and the truth is that our economics should ensure alignment, that quality begets profits and that one can't "hack" the metrics.
Apple is far from alone here though. I'd say this "low hanging infuriating bullshit" is actually quite common. In fact, I think it is becoming more common. I have argued here before about the need for more "grumpy developers." I think if you're not grumpy, you should be concerned. Our job is to find problems, break them down into a way that can be addressed, and to resolve them. The "grumpiness" here is a dissatisfaction with the state of things. Given that nothing is perfect, there should always be reason to be "grumpy." A good developer should be able to identify and fix problems without being asked. But I do think there's a worrying decline of (lack of!) "grumpy" types, and I have no doubt this is connected to the rapid rise of vaporware and shitty products.
Also, I notice you play Devil's advocate a lot. While I think it can be useful, I think it can be overused. It needs to drive home the key limitations to an argument, especially when they are uncomfortable. Though I think in our case, I'm the one making the argument that diverges from the norm.
We as developers are tasked with figuring out what the problem is, especially in cases where the client is wrong about their assumptions
> Building the wrong thing fast is about as useful as not building anything at all.
Fully agree. Plus, you may be faster in the short term but you won't in the long run. The effects of both good code and bad code compound. "Tech debt" is just a fancy term for "compounding shit". And it is true, all code is shit, but it isn't binary; there is a big difference between being stepping in shit and being waist deep in shit.
I can predict some of the responses
Premature optimization is the root of all evil
[0] https://softwareengineering.stackexchange.com/a/80092
[1] Understanding the limitations of big O analysis really helps in understanding why this point matters. Usually when n is small, you can have worse big O and still be faster. But the constants we drop off often aren't a rounding error. https://csweb.wooster.edu/dbyrnes/cs200/htmlNotes/qsort3.htm
Even then though, I asked o1-cursor to start a react app. It failed, mostly because it's out of date. It's instructions were for react 2 versions ago.
This seems like an issue. If the statistically most likley answer is old, that's not helpful.
I might be reading into your comment, but I agree "top-down" development sucks: "Give me a react that does X". I've had much more success going bottom-up.
And I've often seen models getting confused on versions. You need to be explicit, and even then then forget.
why the top comments on HN are always people who have not read the article
Perhaps there's gonna be post-AI programming movement where people actually stare at the same monitor and discuss while one of them is coding.
As a sidenote - we've done experiments with FOBsters, and when paired this way, the multiply their output. There's something about psychology of groups and how one can only provide maximum output when teaming.
Even for solo activities, and non-IT activities, such as skiing/snowboard, it is better to have a partner to ride with you and discuss the terrain.
I'm now bracing for the "oh sht, we're all out of a job next year" narrative.
I don't want an LLM to generate "the answer" for me in a lot of places, but I do think it's amazing for helping me gather information (and cite where that information came from) and pointers in directions to look. A search engine that generates a concrete answer via LLM is (mostly) useless to me. One that gives me an answer and then links to the facts it used to generate that answer is _very_ useful.
It's the same way with programming. It's great helping you find what you need. But it needs to be in a way that you can verify it's right; or take it's answer and adjust it to what you actually need (based on the context it provides).
Maybe. My sense if we'd need to see 3 to 4 orders of magnitude improvements on the current models before we can replace people outright.
I do think we'll see a huge productivity boost per developer over the next few years. Some companies will use that to increase their throughput, and some will use it to reduce overhead.
GitHub Copilot is...not. It doesn't seem to understand how to help me as well as ChatGPT does.
I think chat is a nice intermediary evolution between the CLI (that we use every day) and whatever comes next.
I work at Augment (https://augmentcode.com), which, surprise surprise, is an AI coding assistant. We think about the new modality required to interact with code and AI on a daily basis.
Beside increase productivity (and happiness, as you don't have to do mundane tasks like tests, documentations, etc), I personally believe that what AI can open up is actually more of a way for non-coders (think PMs) to interact with a codebase. AI is really good at converting specs, user stories, and so on into tasks—which today still need to be implemented by software engineers (with the help of AI for the more tedious work). Think of what Figma did between designers and developers, but applied to coding.
What’s the actual "new UI/UX paradigm"? I don’t know yet. But like with Figma, I believe there’s a happy ending waiting for everyone.
That is to say: I think LLMs are going to make a problem we already had (much) worse.
My experiments have been nowhere near that successful.
I would love, love, love to see a transcript of how that process worked over an hour, if that was something you were willing to share.
I do all this + rubber ducky the hell out of it.
Sometimes I just discuss concepts of the project with the thing and it helps me think.
I dont think chat is going to be right for everyone but it absolutely works for me.
It is helpful to frame this in the historical arc described by Yuval Harari in his recent book "Nexus" on the evolution of information systems. We're at the dawn of history for how to work with AI, and actively visualizing the future has an immediate ROI.
"Chat" is cave man oral tradition. It is like attempting a complex Ruby project through the periscope of an `irb` session. One needs to use an IDE to manage a complex code base. We all know this, but we haven't connected the dots that we need to approach prompt management the same way.
Flip ahead in Harari's book, and he describes rabbis writing texts on how to interpret [texts on how to interpret]* holy scriptures. Like Christopher Nolan's movie "Inception" (his second most relevant work after "Memento"), I've found myself several dreams deep collaborating with AI to develop prompts for [collaborating with AI to develop prompts for]* writing code together. Test the whole setup on multiple fresh AI sessions, as if one is running a business school laboratory on managerial genius, till AI can write correct code in one shot.
Duh? Good managers already understand this, working with teams of people. Technical climbers work cliffs this way. And AI was a blithering idiot until we understood how to simulate recursion in multilayer neural nets.
AI is a Rorschach inkblot test. Talk to it like a kindergartner, and you see the intelligence of a kindergartner. Use your most talented programmer to collaborate with you in preparing precise and complete specifications for your team, and you see a talented team of mature professionals.
We all experience degradation of long AI sessions. This is not inevitable; "life extension" needs to be tackled as a research problem. Just as old people get senile, AI fumbles its own context management over time. Civilization has advanced by developing technologies for passing knowledge forward. We need to engineer similar technologies for providing persistent memory to make each successive AI session smarter than the last. Authoring this knowledge helps each session to survive longer. If we fail to see this, we're condemning ourselves to stay cave men.
Compare the history of computing. There was a lot of philosophy and abstract mathematics about the potential for mechanical computation, but our worldview exploded when we could actually plug the machines in. We're at the same inflection point for theories of mind, semantic compression, structured memory. Indeed, philosophy was an untestable intellectual exercise before; now we can plug it in.
How do I know this? I'm just an old mathematician, in my first month trying to learn AI for one final burst of productivity before my father's dementia arrives. I don't have time to wait for anyone's version of these visions, so I computed them.
In mathematics, the line in the sand between theory and computation keeps moving. Indeed, I helped move it by computerizing my field when I was young. Mathematicians still contribute theory, and the computations help.
A similar line in the sand is moving, between visionary creativity and computation. LLMs are association engines of staggering scope, and what some call "hallucinations" can be harnessed to generalize from all human endeavors to project future best practices. Like how to best work with AI.
I've tested everything I say here, and it works.
I had crucial area with threads. Code generated by chat seemed to be ok, but had one flaw. My initial code written manually was bug free. chat-generated output was not. It was difficult to catch it via inspection.
In large, I assert this is because the best way to do something is to do that thing. There can be correspondence around the thing, but the artifacts that you are building are separate things.
You could probably take this further and say that narrative is a terrible way to build things. It can be a great way to communicate them, but being a separate entity, it is not necessarily good at making any artifacts.
Even with game development. Level editors have a good history for being how people actually make games. Some quite good ones, I should add.
For website development, many template based systems worked quite well. People seem hellbent on never acknowledging that form builders of the late 90s did, in fact, work.
Is it a bit nicer that you can do everything through a dialog? I'm sure it is a great for people that think that way.
As is often the case in these sorts of thingsz your milage may vary for the more complex settings.
Incidentally I think that's also a good model for how much to trust the output - you might have a colleague who knows enough about X to think they can answer your question, but they're not necessarily right, you don't blindly trust it. You take it as a pointer, or try the suggestion (but not surprised if it turns out it doesn't work), etc.
People love complaining about the email workflow of git, but it is demonstrably better than any chat program for what it is doing.
I'm thinking of things that are assembled. The correspondence that went into the assembly is largely of historical interest, but not necessarily one of current use.
Is your issue that you want to discuss the thing you are collaborating on outside of the tool you are creating it in?
We have some tools integrated with email to help version control things. But the actual version control is, strictly, not the emails.
I have never had any issue finding information in slack with history going back nearly a decade. The only issue I have with Slack is a people problem where most communication is siloed in private channels and DMs.
Email threads are incredibly hard to follow though. The UX is rough and it shows.
The fact that there's a subject header alone leads people to both stay on topic and have better thought out messages.
I agree that email threads could have better UX. Part of that is the clients insistence on appending the previous message to every reply. This is completely optional though and should probably be turned off by default for simple replies.
Email is really powerful but people simply aren't good at taking advantage of it and it varies by email client. Doing some IT work at a startup made this pretty clear to me. I found Slack was much more intuitive for people.
Both systems rely on the savviness of the users for the best experience and I just think email is losing the UX war. Given how terrible people seem to be at communicating I think it's a pretty important factor to consider.
I've also generally had the opposite experience, a huge amount of business offices live and breath in email (mostly Outlook, but I'm sure it varies). Startups tend to run fast and lean, but as soon as you have some threshold of people, email is king.
I'm not hating on email, it has a lot of good properties and still serves a purpose. Every office appears to have some kind anti-slack vigilante. It's really not that bad.
It doesn't help that Outlook's search capabilities have gotten effectively useless - I can type in search terms that I'm literally looking at in my inbox and have it return no results, or have it return dozens of hits without the search terms involved at all. I don't have that problem with Slack or Teams.
However, I think you are right overall on email being better overall for what people end up using chat apps for.
I wish I could block them within all these chat apps.
"Sorry, you can't bother to send voice messages to this person."
The last counter argument I read got buried on Discord or Slack somewhere.
Here, ephemeral means "this conversation might as well never had happened", so why waste time on that?
Does that mean I can't have some pleasure in conversing about things? Of course not. But, I also enjoy some pleasure there from the low stakes and value that a conversation has. It should be safe to be wrong. If you have a conversation spot where being wrong is not safe, then I question what is the advantage of that over trying to adopt a legalese framework for all of your communication?
So now imagine such (idealized) HN threads transplanted to Discord or Slack. Same people, same topics, same insights, just unrolling in the form of a regular chat. All that value, briefly there to partake in, and then forever lost after however much time it takes for it to get pushed up a few screens worth of lines in the chat log. People don't habitually scroll back very far on a regular basis (and the UI of most chat platforms starts to rapidly break down if you try), and the lack of defined structure (bounded conversations labeled by a topic) plus weak search tools means you're unlikely to find a conversation again even if you know where and when it took place.
That, plus ephemeral nature of casual chat means not just the platform, but also some of the users expect it to quickly disappear, leading to what I consider anti-features such as the ability to unilaterally edit or unsend any message at arbitrary time in the future. It takes just one participant deciding, for whatever reason, to mass-delete their past messages, for many conversations to lose most of their value forever.
--
[0] - Especially that the traditional communication style, both private and business, is overly verbose. Quite like a chat, in fact, but between characters in a theatrical play - everyone has longer lines.
I think my mental model is more for chat rooms to take the place of coffee room chats. Ideally, some of those do push on something to happen. I'm not sure that forcing them into the threaded structure of conversations really helps, though?
Maybe it is based on the aim? If the goal is a simulacrum of human contact, then I think ephemeral makes a ton of sense.
I also kind of miss the old tradition of having a "flamewars" topic in newsgroups. I don't particularly love yelling at each other, but I do hate that people can't bring up some topics.
(I also miss some old fun newsgroups. I recall college had Haiku and a few other silly restrictive style groups that were just flat fun.)
I also confess this model of ephemeral conversation is amusing in this specific website. Which I also largely view as a clubhouse conversation that is also best viewed as ephemeral. But it is clearly held for far longer than that idea would lead me to think.
While all of these features could in principle be realized in a chat system as well, in practice they don’t provide that flexibility and power.
Another usability feature of emails is that they have a subject line. This allows to meaningfully list emails in a compact fashion. In a desktop interface, you can easily view and visually grep 50 emails or more at once in a mail folder or list of search results (in something like Outlook or Thunderbird or Mutt). This allows working with emails more efficiently than with a chat view where you can only see a few messages at once, and only of the same thread or channel.
Yet another usability feature of emails is that each email has its own read/unread status. This, again, is facilitated by each email being its own separate data object, and by the separation between subject and body, which allows the read status to be unambiguously bound to “opening” the email, or to navigating in the list of emails alongside a preview pane. And you can mark any email as unread again. In chats, the granularity of read/unread is the whole chat, whether you’ve actually read all of it or not. You can’t easily track what you’ve read or not in an automated way as with email, other than by that coarse-grained linear time-based property of when you last visited the channel.
My experience with email is that people have subject lines, email explicitly identifies to and cc recipients; email is threaded; email often has quotes/excerpting/highlighting from prior parts of the thread.
On the other hand, most chat usage I see is dependent on temporal aspects for threading (people under-utilize platform features for replies etc), tagging is generally only done to ping people to attract attention, chat groups are frequently reused for multiple different purposes.
Leaping to a point-in-time within a chat stream is often a bad user experience, with having to scroll up and down through unrelated stuff to find what you’re looking for.
Stuff in email is just massively more discoverable for me.
This obvious assumes that who ever wrote the email isn't a madman, who insist on using emails like it was a chat.
Funnily, this is the exact same issue I have with Slack.
This is why, if something is important, I take it out of email and put it into a document people can reference. The latest and correct information from all the decisions in the thread can also be collected in one place, so everyone reading doesn’t have to figure it out. Not to mention side conversations can influence the results, without being explicitly stated in the email thread.
This is how things should be done, regardless of which medium is used to discuss the project. Without isolating and aggregating the final decision of each thread, there is no way to determine what everybody has agreed upon as the final product without looking back, which quickly becomes onerous.
Things get messy when you start having different versions of each feature, but that doesn't change the concept of using email/Slack/Discord/text/etc. for discussion and a separate "living" document for formalizing those decisions.
I’ve been saying this since 2018
You have some of the same problems with email, of course. Losing threading, in particular, made things worse. It was a "chatification of email" that caused people to lean in to email being bad. Amusing that we are now seeing chat applications rise to replace email.
I'm actually in an awkward position where I was very supportive of RTO two years ago, but have since become very reliant on some things I could not do with a rigid RTO policy.
Regardless of RTO or WFH, patience and persistence remain vital qualities.
Talking to a computer still sucks as an user interface - not because a computer can't communicate on multiple channels the way people do, as it can do it now too. It sucks for the same reason talking to people sucks as an user interface - because the kind of tasks we use computers for (and that aren't just talking with/to/at other people via electronic means) are better handle by doing than by talking about them. We need an interface to operate a tool, not an interface to an agent that operates a tool for us.
As an example, consider driving (as in, realtime control - not just "getting from point A to B"): a chat interface to driving would suck just as badly as being a backseat driver sucks for both people in the car. In contrast, a steering wheel, instead of being a bandwidth-limiting indirection, is an anti-indirection - not only it lets you control the machine with your body, the control is direct enough that over time your brain learns to abstract it away, and the car becomes an extension of your body. We need more of tangible interfaces like that with computers.
The steering wheel case, of course, would fail with "AI-level smarts" - but that still doesn't mean we should embrace talking to computers. A good analogy is dance - it's an interaction between two independently smart agents exploring an activity together, and as they do it enough, it becomes fluid.
So dance, IMO, is the steering wheel analogy for AI-powered interfaces, and that is the space we need to explore more.
Excellent comment and it gets to the heart of something I've had trouble clearly articulating: We've slowly lost the concept that a computer is a tool that the user wields and commands to do things. Now, a computer has its own mind and agency, and we "request" it to do things and "communicate" with it, and ask it to run this and don't run that.
Now, we're negotiating and pleading with the man inside of the computer, Mr. Computer, who has its own goals and ambitions that don't necessarily align with your own as a user. It runs what it wants to run, and if that upsets you, user, well tough shit! Instead of waiting for a command and then faithfully executing it, Mr. Computer is off doing whatever the hell he wants, running system applications in the background, updating this and that, sending you notifications, and occasionally asking you for permission to do even more. And here you are as the user, hobbled and increasingly forced to "chat" with it to get it to do what you want.
Even turning your computer off! You used to throw a hardware switch that interrupts the power to the main board, and _sayonara_ Mr. Computer! Now, the switch does nothing but send an impassioned plea to the operating system to pretty please, with sugar on top, when you're not busy could you possibly power off the computer (or mostly power it off, because off doesn't even mean off anymore).
The modern OS values the system's theoretical 'system health' metrics far above things like "whether the user can use it to do some user task."
Another great example is how you can't boot a modern Mac laptop, on AC power, until it has decided its battery is sufficiently charged. Why? None of your business.
Anyway to get back on topic, this is an interesting connection you've made, the software vendor will perhaps delegate decisions like "is the user allowed to log into the computer at this time" or "is a reboot mandatory" to an "agent" running on the computer. If we're lucky we'll get to talk to that agent to plead our case, but my guess is Apple and Microsoft will decide we aren't qualified to have input to the decisions.
System Integrity Protection is designed to allow modification of these protected parts only by processes that are signed by Apple and have special entitlements to write to system files, such as Apple software updates and Apple installers.
FWIW this happens what happens with modern steering wheels as well. Power steering is its own complicated subsystem that isn't just about user input. It has many more failure modes than an old-fashioned, analog steering wheel. The reason folks feel like "Mr. Computer" has a mind of its own is because of the mismatch between user desire and effect. This is a UX problem.
I also think chat and RAG are the biggest two UX paradigms we've spent exploring when it comes to LLMs. It's probably worth folks exploring other UX for LLMs that are enabling for the user. Suggestions in documents and code seem to be a UX that more people enjoy using but even then there's a mismatch.
But, as you say, a chat interface would be a terrible way to actively drive a car. And that is a different thing, but I'm growing convinced many will focus on the first idea while staving off the complaints of the latter.
In another thread, I assert that chat is probably a fine way to order up something that fits a repertoire that trained a bot. But, I don't think sticking to the chat window is the best way to interface with what it delivers. You almost certainly want to be much more actively "hands on" in very domain specific ways with the artifacts produced.
Yes, this is what I've also tried to hint at in my comment, but failed part-way. In most of the cases I can imagine chat interface to be fine (or even ideal), it's really only good as a starting point. Take two examples based on your reply:
1) Getting a car ride. "Computer, order me a cab home" is a good start. It's even OK if I then get asked to narrow it down between several different services/fares (next time I'll remember to specify that up front). But if I want to inspect the route (or perhaps adjust it, in a hypothetical service that supports it), I'd already prefer an interactive map I can scroll and zoom, with PoIs I can tap on to get their details, than to continue a verbal chat.
2) Ordering food in a fast food restaurant. I'm fine starting it with a conversation if I know what I want. However, getting back the order summary in prose (or worse, read out loud) would already be taxing, and if I wanted to make final adjustments, I'd beg for buttons and numeric input boxes. And, in case I don't know what I want, or what is available (and at what prices), a chat interface is a non-starter. Interactive menu is a must.
You sum this up perfectly:
> You almost certainly want to be much more actively "hands on" in very domain specific ways with the artifacts produced.
Chat may be great to get that first artifact, but afterwards, there's almost always a more hands-on interface that would be much better.
EDIT: love your analogy to dance!
It's a CLI without the integrity. When you bought a 386, it came with a big book that said "MS-DOS 4.01" and enumerated the 75 commands you can type at the C:\> prompt and actually make something useful happen.
When you argue with ChatGPT, its whole business is to not tell you what those 75 commands are. Maybe your prompt fits its core competency and you'll get exactly what you wanted. Maybe it's hammering what you said into a shape it can parse and producing marginal garbage. Maybe it's going to hallucinate from nothing. But it's going to hide that behind a bunch of cute language and hopefully you'll just keep pulling the gacha and blaming yourself if it's not right.
I really don't like the idea of chatting with an AI though. There are better ways to interface with AIs and the focus on chat is making people forget that.
- "App builders" that use some combination of drag&drop UI builders, and design docs for architecture, workflows,... and let the UI guess what needs to be built "under the hood" (a little bit in the spirit of where UML class diagrams were meant to take us). This would still require actual programming knowledge to evaluate and fix what the bot has built
- Formal requirement specification that is sufficiently rigorous to be tested against automatically. This might go some way towards removing the requirement to know how to code, but the technical challenge would simply shift to knowing the specification language
TLDR: Targeted edits and prompts / Heads Up Display
It should probably be more like an overlay (and hooked into context menus with suggestions, inline context bubbles when you want more context for a code block) and make use of an IDE problems view. The problems view would have to be enhanced to allow it to add problems that spanned multiple files, however.
Probably like the Rust compiler output style, but on steroids.
There would likely be some chatting required, but it should all be at a particular site in the code and then go into some history bank where you can view every topic you've discussed.
For authoring, I think an interactive drawing might be better, allowing you to click on specific areas and then use shorter phrasing to make an adjustment instead of having an argument in some chat to the left of your screen about specificity of your request.
Multi-point / click with minimal prompt. It should understand based on what I clicked what the context is without me having to explain it.
Ok, but what is a good pattern to leverage AI tools for coding?
The space is ripe for folks with actual domain expertise to design an appropriate AI workflow for their domain.
But, if you want to start doing "domain specific" edits to the artifacts that are made, you are almost certainly going to want something like the app builders idea. Down thread, I mention how this is a lot like procedural generative techniques for game levels and such. Such that I think I am in agreement with your first bullet?
Similarly, if you want to make music with an instrument, it will be hard to ignore playing with said instrument more directly. I suspect some people can create things using chat as an interface. I just also suspect directly touching the artifacts at play is going to be more powerful.
I think I agree with the point on formal requirements. Not sure how that really applies to chat as an interface? I think it is hoping for a "laws of robotics" style that can have a test to confirm them? Reality could surprise me, but I always viewed that as largely a fiction item.
It's usually not. Narrative is a famously flawed way to communicate or record the real world.
It's great for generating engagement, though.
...and yet with it's flaws, it's the most flexible in conveying meaning. A Ted Chiang interview was on the HN frontpage a few days ago, in it, he mentions that humans created multiple precise, unambiguous communication modes like equations used in mathematical papers and proofs. But those same papers are not 100% equations, the mathematicians have to fall back to flawed language to describe and provide context because those formal languages only capture a smaller range of human thought compared to natural language.
This is not to say chat has the best ergonomics for development - it's not, but one has to remember that the tools are based on Large Language Models whose one-trick is manipulating language. Better ergonomics would likely come from models trained or fine-tuned on AST-tokens and diffs. They'd still need to modulate on language (understanding requirements, hints, variable names,and authoring comments, commits and/or PRs).
Chat is a great UI pattern for ephemeral conversation. It's why we get on the phone or on DM to talk with people while collaborating on documents, and don't just sit there making isolated edits to some Google Doc.
It's great because it can go all over the place and the humans get to decide which part of that conversation is meaningful and which isn't, and then put that in the document.
It's also obviously not enough: you still need documents!
But this isn't an "either-or" case. It's a "both" case.
I created the tetr app[1] which is basically “chat UI for everything”. I did that because I used to message myself notes and wanted to expand it to many more things. There’s not much back and forth, usually 1 input and instant output (no AI), still acting like a chat.
I think there’s a lot of intuitiveness with chat UI and it can be a flexible medium for sharing different information in a similar format, minimizing context switching. That’s my philosophy with tetr anyhow.
I think this applies to any “fuzzy generation” scenario. It certainly shouldn’t be the only tool, and (at least as it stands today) isn’t good enough to finalize and fine tune the final result, but a series of “a foo with a bar” “slightly less orange” “make the bar a bit more like a fizzbuzz” interactions with a good chat UI can really get a good 80% solution.
But like all new toys, AI and AI chat will be hammered into a few thousand places where it makes no sense until the hype dies down and we come up with rules and guidelines for where it does and doesn’t work
I heavily disagree here, chat - or really text - is a horrible UI for image generation, unless you have almost zero idea of what you want to achieve and you don't really care about the final results.
Typing "make the bar a bit more like a fizzbuzz" in some textbox is awful UX compared to, say, clicking on the "bar" and selecting "fizzbuzz" or drag-and-dropping "fizzbuzz" on the "bar" or really anything that takes advantage of the fact we're interacting with a graphical environment to do work on graphics.
In fact it is a horrible UI for anything, except perhaps chatbots and tasks that have to do with text like grammar correction, altering writing styles, etc.
It is helpful for impressing people (especially people with money) though.
That assumes that you have a UX capable of determining what you're clicking on in the generated image (which we could call a given if we assume a sufficiently capable AI model since we're already instructing it to alter the thing), and also that it can determine from your click that you've intended to click on the "foo" not the "greeble" that is on the foo or the shadow covering that part of the foo or anything else that might be in the same Z stack as your intended target. Pixel bitching adventure games come to mind as an example of how badly this can go for us. And yes, this is solvable, Squeak has a UI where repeatedly clicking in the same spot will iterate through the list of possibilities in that Z stack. But it could also get really messy really quickly.
Then we have to assume that your UX will be able to generate an entire list of possible things you might want to be able to to do with that thing that you've clicked, including adding to it, removing it, removing part of it, moving it, transforming its dimensions, altering its colors, altering the material surface and on and on and on. And that list of possibilities needs to be navigable and searchable in a way that's faster than just typing "make the bar more like a fizzbuzz" into a context aware chat box.
Again, I'm not arguing the chat interface should be the only interface. In fact, as you point out we're using a graphical system, it would be great if you could click on things or select them and have the system work on too. It should be able to take additional input than just chat. But I still think for iterating on a fuzzy idea, a chat UI is a useful tool.
About how the exact UI will be for multiple objects, this is something that multiple professional tools - like 3D engines or editors - already do. A common solution (aside from the multiple click you mentioned) is to use a modifier key like Alt+click to show a list of the items under the mouse. An AI system could also show alternative interpretations of what is clicked.
> Again, I'm not arguing the chat interface should be the only interface.
The main issue here is that current AIs are trained using chat-like interactivity at their core with everything else being a "wrapper" around the chat parts, when it should be the other way around.
I think it is brilliant. On another hand I caught myself many times writing prompts to colleagues. Although it made requirements of what I need so much clearer for them.
Agreed that copy pasting context in and out of ChatGPT isn't the fastest workflow. But Cursor has been a major speed up in the way I write code. And it's primarily through a chat interface, but with a few QOL hacks that make it way faster:
1. Output gets applied to your file in a git-diff style. So you can approve/deny changes.
2. It (kinda) has context of your codebase so you don't have to specify as much. Though it works best when you explicitly tag files ("Use the utils from @src/utils/currency.ts")
3. Directly inserting terminal logs or type errors into the chat interface is incredibly convenient. Just hover over the error and click the "add to chat"
I’ve only been slowed down with AI tools. I tried for a few months to really use them and they made the easy tasks hard and the hard tasks opaque.
But obviously some people find them helpful.
Makes me wonder if programming approaches differ wildly from developer to developer.
For me, if I have an automated tool writing code, it’s bc I don’t want to think about that code at all.
But since LLMs don’t really act deterministically, I feel the need to double check their output.
That’s very painful for me. At that point I’d rather just write the code once, correctly.
Leaning on these tools just isn’t for me rn.
I like them most for one off scripts or very small bash glue.
I feel the same
> That’s very painful for me. At that point I’d rather just write the code once, correctly.
I use AI tools augmentatively, and it's not painful for me, perhaps slightly inconvenient. But for boiler-plate-heavy code like unit tests or easily verifiable refactors[1], adjusting AI-authored code on a per-commit basis is still faster than me writing all the code.
1. Like switching between unit-test frameworks
Scripting assistance by itself is worth the price of admission.
The other thing I've found it good at is giving me an English description of code I didn't write... I'm sure it sometimes hallucinates, but never in a way that has been so wrong that its been apparent to me.
Very good for throwaway code though, for example a PoC which won't really be going to production (hopefully xD).
Maybe it’s bc I’ve been programming since I was young or because I mainly learned by doing code-along books, but writing the code is where my thinking gets done.
I don’t usually plan, then write code. I write code, understand the problem space, then write better code.
I’ve known friends and coworkers who liked to plan out a change in psudocode or some notes before getting into coding.
Maybe these different approaches benefit from AI differently.
Although most recently I caught it because I fed it into both gpt-4o and o1 and o1 had the correct flags. Then I asked 4o to expand the flags from the short form to the long form and explain them so I could double-check my reasoning as to why o1 was correct.
It was already wrong once.
The chat interface is... fine. Certainly better integrated into the editor than GitHub Copilot's, but I've never really seen the need to use it as chat—I ask for a change and then it makes the change. Then I fixed what it did wrong and ask for another change. The chat history aspect is meaningless and usually counterproductive, because it's faster for me to fix its mistakes than keep everything in the chat window while prodding it the last 20% of the way.
"how do I check the cors bucket policies on [S3 bucket name]"
hint: you don't get paid to get the LLM to output perfect code, you get paid by PRs submitted and landed. Generate the first 80% or whatever with the LLM, and then finish the last 20% that you can write faster than the LLM yourself, by hand.
End users don’t care where the code came from.
https://zed.dev/docs/assistant/assistant-panel#editing-a-con...
It's still a "chat" but it's just text at the end of the day. So you can edit as you see fit to refine your context and get better responses.
Cursor also does a great job of showing inline diffs of what composer is doing, so you can quickly review every change.
I don’t think there’s any reason Continue can’t match these features, but it hadn’t, last I checked.
Cursor also focuses on sane defaults, which is nice. The tab completion model is very good, and the composer model defaults to Claude 3.5 Sonnet, which is arguably the best non-reasoning code model. (One would hope that Cursor gets agent-composer working with reasoning models soon.) Continue felt much more technical… which is nice for power users, but not always the best starting place.
Imperative: - write a HTTP server that serves jokes - add a healthcheck endpoint - add TLS and change the serving port to 443
Declarative: - a HTTP server that serves jokes - contains a healthcheck endpoint - supports TLS on port 443
The differences here seem minimal because you can see all of it at once, but in the current chat paradigm you'd have to search through everything you've said to the bot to get the full context, including the side roads that never materialized.
In the document approach you're constantly refining the document. It's better than reviewing the code because (in theory) you're looking at "support TLS on port 443" instead of a lot of code, which means it can be used by a wider audience. And ideally I can give the same high level spec to multiple LLMs and see which makes the best application.
For example, for a signup page, we could have: - Declarative: Signup the user using their email address - Imperative: To do the same, we will need to implement the smtp library, which means discovering that we need an SMTP server, so now we need to choose which one. And when purchasing an SMTP Server plan, we discover that there are rate limit, so now we need to add some bot protection to our signup page (IP Rate Limit only? ReCaptcha? Cloudflare bot protection?), etc
Which means that at the end, the imperative code way is kind of like the ultimate implementation specs.
The source of truth would still be the code though, otherwise the declarative statements would get so verbose that they wouldn't be any more useful than writing the code itself.
e.g. executives treat the org as a blackbox LLM and chat w it to get real results
I found interacting with it via chat to be super-useful and a great way to get stuff done. Yeah, sometimes you just have to drop into the code, and tag a particular line and say "this isn't going to work, rewrite it to do x" (or rewrite it yourself), but the ability to do that doesn't vitiate the value of the chat.
Zed makes it trivial to attach documentation and terminal output as context. To reduce risk of hallucination, I now prefer working in static, strongly-typed languages and use libraries with detailed documentation, so that I can send documentation of the library alongside the codebase and prompt. This sounds like a lot of work, but all I do is type "/f" or "/t" in Zed. When I know a task only modifies a single file, then I use the "inline assist" feature and review the diffs generated by the LLM.
Additionally, I have found it extremely useful to actually comment a codebase. LLMs are good at unstructured human language, it's what they were originally designed for. You can use them to maintain comments across a codebase, which in turn helps LLMs since they get to see code and design together.
Last weekend, I was able to re-build a mobile app I made a year ago from scratch with a cleaner code base, better UI, and implement new features on top (making the rewrite worth my time). The app in question took me about a week to write by hand last year; the rewrite took exactly 2 days.
---
As a side note: a huge advantage of Zed with locally-hosted models is that one can correct the code emitted by the model and force the model to re-generate its prior response with those corrections. This is probably the "killer feature" of models like qwen2.5-coder:32b. Rather than sending extra prompts and bloating the context, one can just delete all output from where the first mistake was made, correct the mistake, then resume generation.
So you either need lots of extra text to remove the ambiguity of natural language if you use AI or you need a special precise subset to communicate with AI and that’s just programming with extra steps.
Joking aside, this is likely where we will end up, just with a slightly higher programming interface, making developers more productive.
In other words, if you replace natural language with a programming language then the computer will do a good job of interpreting your intent. But that's always been true, so...
An LLM can do increasingly well as a fly on the wall, but it’s common for people using an LLM to be less collaborative with an LLM and for them to expect the LLM to structure the conversation. Hence the suggestion to be careful in your prompting.
Right. On one side you have programming language and on the other natural language.
They can intermingle, if that is what you are trying to say? You can see this even in traditional computer programming. One will often switch between deliberate expression and casual, natural expression (what often get called comments in that context).
In other word, complex applications can still be fully specified in plain English, even if it might take more words.
In plain English, of course, but not in natural English. When using language naturally one will leave out details, relying on other inputs, such as shared assumptions, to fill in the gaps. Programming makes those explicit.
Likewise for English: one can use natural English to add as many details as necessary, depending on who you're talking to, e.g. "Make an outline around the input field, and color the outline #ff0000." You can then add, if necessary, "Make the corners of the outline rounded with a 5 pixel radius."
In this respect, complex applications can be fully specified in English; we usually call those documents "formal specifications." You can write it terse, non-natural language with consistent, defined terminology to save room (as most specs are), or colloquial (natural) language if you really want. I wouldn't recommend the latter, but it's definitely useful when presenting specs to a less technically informed audience.
Of course. We established that at the beginning. The entire discussion is about exactly that. It was confirmed again in the previous comment. However, that is not natural. I expect most native English speakers would be entirely incapable of fully specifying a complex application or anything else of similar complexity. That is not natural use.
While the words, basic syntax, etc. may mirror that found in natural language, a specification is really a language of its own. It is nothing like the language you will find people speaking at the bar or when writing pointless comments on Reddit. And that's because it is a programming language.
Your original postulation was that it simply wasn't possible, implying nobody could do it. The fact that most native English speakers wouldn't be able to do it doesn't mean nobody can do it.
I agree that most native English speakers wouldn't be able to write a reasonably complete spec in any type of language, not just because they lack the language skill, but because they simply wouldn't have the imagination and knowledge of what to create to begin with, let alone how to express it.
Then you must have mistakenly replied to the wrong comment, I guess? My original comment, and every one that followed, postulated that so-called "careful use of English", as in what you are talking about and what we have always been talking about, is a programming language. Given that it is a programming language, how could it not be used in that way? That's what programming languages do best.
> The fact that most native English speakers wouldn't be able to do it doesn't mean nobody can do it.
Of course. But the fact that most native English speakers wouldn't be able to do it proves that it is isn't natural language. This "technical English" language may resemble the natural language also known as English in many ways, but, as even you pointed out earlier, it is not the same language.
They all usually build down to a subset of english, because near caveman speak is enough to define things with precision.
Which model are you talking about here? Because with ChatGPT, I struggle with getting it to ask any clarifying questions before just dumping code filled with placeholders I don't want, even when I explicitly prompt it to ask for clarification.
Real projects don't require an infinitely detailed specification either, you usually stop where it no longer meaningfully moves you towards the goal.
The whole premise of AI developer automation, IMO, is that if a human can develop a thing, then AI should be able too, given the same input.
With a 200k token window like Claude has you can already dump a lot of design docs / transcripts / etc. at it.
If you use cline with any large context model the results can be pretty amazing. It's not close to self guiding, You still need to break down and analyze the problem and provide clear and relevant instructions. IE you need to be a great architect. Once you are stable on the direction, its awe inspiring to watch it do the bulk if the implementation.
I do agree that there is space to improve over embedded chat windows in IDEs. Solutions will come in time.
By the way, remind me why you need design meetings in that ideal world?:)
> Real projects don't require an infinitely detailed specification either, you usually stop where it no longer meaningfully moves you towards the goal.
The point was that specification is not detailed enough in practice. Precise enough specification IS code. And the point is literally that natural language is just not made to be precise enough. So you are back where you started
So you waste time explaining in detail and rehashing requirements in this imprecise language until you see what code you want to see. Which was faster to just... idk.. type.
Having a feedback loop is the only way viable for this. Sure, the client could give you a book on what they want, but often people do not know their edge cases, what issues may arise/etc.
All the same buzzwords, including "AI"! In 1981!
If you know how to program, then I agree and part of why I don't see the point. If you don't know how to program, than the prompt isn't much different than providing the specs/requirements to a programmer.
haha, I just imagined sending TypeScript to ChatGPT and having it spit my TypeScript back to me. "See guys, if you just use Turing-complete logically unambiguous input, you get perfect output!"
The struggle is to provide a context that disambiguates the way you want it to.
LLMs solve this problem by avoiding it entirely: they stay ambiguous, and just give you the most familiar context, letting you change direction with more prompts. It's a cool approach, but it's often not worth the extra steps, and sometimes your context window can't fit enough steps anyway.
My big idea (the Story Empathizer) is to restructure this interaction such that the only work left to the user is to decide which context suits their purpose best. Given enough context instances (I call them backstories), this approach to natural language processing could recursively eliminate much of its own ambiguity, leaving very little work for us to do in the end.
Right now my biggest struggle is figuring out what the foundational backstories will be, and writing them.
const a = “abcd”
The advantage of natural language is that we can write ambiguously defined expressions, and infer their meaning arbitrarily with context. This means that we can write with fewer unique expressions. It also means that context itself can be more directly involved in the content of what we write.
In context-free grammar, we can only express "what" and "how"; never "why". Instead, the "why" is encoded into every decision of the design and implementation of what we are writing.
If we could leverage ambiguous language, then we could factor out the "why", and implement it later using context.
I think about this like SQL in the late 80s. At the time, SQL was the “next big thing” that was going to mean we didn’t need programmers, and that management could “write code”. It didn’t quite work out that way, of course, as we all know.
I see chat-based interfaces to LLMs going exactly the same way. The LLM will move down the stack (rather than up) and much more appropriate task-based UX/UI will be put on top of the LLM, coordinated thru a UX/UI layer that is much sympathetic to the way users actually want to interact with a machine.
In the same way that no end-users ever touch SQL these days (mostly), we won’t expose the chat-based UX of an LLM to users either.
There will be a place for an ad-hoc natural language interface to a machine, but I suspect it’ll be the exception rather than the rule.
I really don’t think there are too many end users who want to be forced to seduce a mercurial LLM using natural language to do their day-to-day tech tasks.
Only when someone discovers another paradigm that matches or exceeds the effectiveness of LLMs without being a language model.
If you asked me two or three years ago I would have strongly agreed with this theory. I used to point out that every line of code was a decision made by a programmer and that programming languages were just better ways to convey all those decisions than human language because they eliminated ambiguity and were much terser.
I changed my mind when I saw how LLMs work. They tend to fill in the ambiguity with good defaults that are somewhere between "how everybody does it" and "how a reasonably bright junior programmer would do it".
So you say "give me a log on screen" and you get something pretty normal with Username and Password and a decent UI and some decent color choices and it works fine.
If you wanted to provide more details, you could tell it to use the background color #f9f9f9, but a part of what surprised my and caused me to change my mind on this matter was that you could also leave that out and you wouldn't get an error; you wouldn't get white text on white background; you would get a decent color that might be #f9f9f9 or might be #a1a1a1 but you saved a lot of time by not thinking about that level of detail and you got a good result.
Right now we have a ton of AI/ML/LLM folks working on this first clear challenge: better models that generate better defaults, which is great—but also will never solve the problem 100%, which is the second, less-clear challenge: there will always be times you don't want the defaults, especially as your requests become more and more high-level. It's the MS Word challenge reconstituted in the age of LLMs: everyone wants 20% of what's in Word, but it's not the same 20%. The good defaults are good except for that 20% you want to be non-default.
So there need to be ways to say "I want <this non-default thing>". Sometimes chat is enough for that, like when you can ask for a different background color. But sometimes it's really not! This is especially true when the things you want are not always obvious from limited observations of the program's behavior—where even just finding out that the "good default" isn't what you want can be hard.
Too few people are working on this latter challenge, IMO. (Full disclosure: I am one of them.)
In your example, the issue is not with writing the logon screen (You can find several example on github and a lot of css frameworks have form snippets). The issue is making sure that it works and integrate well with the rest of the project, as well as being easy to maintain.
Something like tldraw's "make real" [1] is a much better bet, imo (not that it's mutually exclusive). Draw a rough mockup of what you want, let AI fill in the details, then draw and write on it to communicate your changes.
We think multi-modally; why should we limit the creative process to just text?
[1] https://tldraw.substack.com/p/make-real-the-story-so-far
Here is an example of our approach:
https://blog.codesolvent.com/2024/11/building-youtube-video-...
We are also using the requirements to build a checklist, the AI generates the checklist from the requirements document, which then serves as context that can be used for further instructions.
Here's a demo:
The mode that I've found most fruitful when using Cursor is treating it almost exactly as I would a pair programming partner. When I start on a new piece of functionality I describe the problem and give it what my thoughts are on a potential solution and invite feedback. Sometimes my solution is the best. Sometimes the LLM had a better idea and frequently we take a modified version of what one of us suggested. Just as you would with a human partner. The result of the discussion is better than what either of us would have done on their own.
I also will do classical ping-pong style tdd with it one we agreed on an approach. I'll write a test; llm makes it pass and write the next test which I'll make pass and so on.
As with a real pair, it's important to notice when they are struggling and help them or take over. You can only do this if you stay fully engaged and understand every line. Just like when pairing. I've found llms get frequently in a loop where something doesn't work and they keep applying the same changes they've tried before and it never works. Understand what they are trying to do and help them out. Don't be a shitty pair for your llm!
It gets even funner when you try to get other models to fix whatever is broken and they too get caught in the same loop. I’ll be like “nope! Your buddy ChatGPT said the same thing and got stuck in such and such loop. Clearly whatever you are trying isn’t working so step back and focus on the bigger picture. Are we even doing this the right way in the first place?”
And of course it still walks down the loop. So yeah, better be ready to fix that problem yourself cause if they all do the same thing you are either way off course or they are missing something!
It will slowly grow in complexity, strictness, and features, until it becomes a brand-new programming language, just with a language model and a SaaS sitting in the middle of it.
A startup will come and disrupt the whole thing by simply writing code in a regular programming language.
> Looking for a low level engineer, who works close to the metal, will work on our prompts
https://aider.chat/docs/usage/watch.html
How jarring it is & how much it takes you out of your own flow state is very much dependent on the model output quality and latency still, but at times it works rather nicely.
I think it's more ideal to have the LLM map text to some declarative pseudocode that's easy to read which is then translated to code.
The example given by Daniel might map to something like this:
define sign-in-screen:
panel background "#f9f9f9":
input email required: true, validate-on-blur: true
input password required: true
button "Sign in" gradient: ("#EEE" "#DDD")
connect-to-database
It decided to output something JSON and maybe YAML once.
The example shows "Sign-in screen" with 4 (possibly more) instructions. This could equivalently have been entered one at a time into 'chat'. If the response for each was graphic and instantaneous, chat would be no worse than non-chat.
What makes non-chat better is that the user puts more thought into what they write. I do agree for producing code Claude with up-front instructions beats ChatGPT handily.
If OTOH AI's actually got as good or better than humans, chat would be fine. It would be like a discussion in Slack or PR review comments.
1) The first thing to improve chats as a genre of interface, is that they should all always be a tree/hierarchy (just like Hacker News is), so that you can go back to ANY precise prior point during a discussion/chat and branch off in a different direction, and the only context the AI sees during the conversation is the "Current Node" (your last post), and all "Parent Nodes" going back to the beginning. So that at any time, it's not even aware of all the prior "bad branches" you decided to abandon.
2) My second tip for designs of Coding Agents is do what mine does. I invented a 'block_begin/block_end' syntax which looks like this, and can be in any source file:
// block_begin MyAddNumbers
var = add(a, b)
return a + b
// block_end
With this syntax you can use English language to explain and reason about extremely specific parts of your code with out expecting the LLM to "just understand". You can also direct the LLM to only edit/update specific "Named Blocks", as I call them.
So a trivial example of a prompt expression related to the above might be "Always put number adding stuff in the MyAddNumbers Block".
To explain entire architectural aspects to the LLM, these code block names are extremely useful.
Proper context is absolutely everything when it comes to LLM use
OpenAI finally made it where you can go back and edit a prior response, in their chat view, but their GUI is jank, because it's not a tree.
I've tested a few integrated AI dev tools and it works like a charm. I don't type all my instructions at once. I do it the same way as I do it with code. Iteratively:
1) Create a layout
2) Fill left side
3) Fill right side
4) Connect components
5) Populate with dummy data
> The first company to get this will own the next phase of AI development tools.
There's more than 25 working on this problem and they are already in production and some are really good.
Everything else, is just putting layers, that are not nearly as capable at an LLM, between me and the raw power of the LLM.
The core realization I made to truly unlock LLM code assistance as a 10x + productivity gain, is that I am not writing code anymore, I am writing requirements. It means being less an engineer, and more a manager, or perhaps an architect. It's not your job to write tax code anymore, it's your job to describe what the tax code needs to accomplish and how it's success can be defined and validated.
Also, it's never even close to true that nobody uses LLMs for production software, here's a write-up by Google talking about using LLMs to drastically accelerate the migration of complex enterprise production systems: https://arxiv.org/pdf/2501.06972
Last night I wrote an implementation of an AI paper and it was so much easier to just discard the automatic chat formatting and do it "by hand": https://github.com/Xe/structured-reasoning/blob/main/index.j...
I wonder if foundation models are an untapped goldmine in terms of the things they can do, but we can't surface them to developers because everyone's stuck in the chat pattern.
Would you be so kind as to ELI5 what you did in that index.js?
I've used ollama to run models locally, but I'm still stuck in chat-land.
Of course, if a blog post is in the works, I'll just wait for that :)
AI models fundamentally work on the basis of "given what's before, what comes next?" When you pass messages to an API like:
[
{ "role": "system", content": "You are an expert in selling propane and propane accessories. Whenever someone talks about anything that isn't propane, steer them back." },
{ "role": "user", "content": "What should I use to cook food on my grill?" },
{ "role": "assistant", "content": "For cooking food on your grill, using propane is a great choice due to its convenience and efficiency. [...]" }
]
You are an expert in selling propane and propane accessories. Whenever someone talks about anything that isn't propane, steer them back.
<｜User｜>What should I use to cook food on my grill?<｜endofsentence｜>
<｜Assistant｜>
For cooking food on your grill, using propane is a great choice due to its convenience and efficiency. [...]<｜endofsentence｜>
What I'm doing here is manually assembling the context window such that I can take advantage of that and then induce the model that it needs to think more, so the basic context window looks like:
Follow this JSON schema: [omitted for brevity]
<｜User｜>Tell me about Canada.<｜endofsentence｜>
<｜Assistant｜><think>Okay
The paper s1: Simple test-time scaling (https://arxiv.org/abs/2501.19393) concludes that this is probably how OpenAI implemented the "reasoning effort" slider for their o1 API. My index.js file applies this principle and has DeepSeek's Qwen 2.5 32b reasoning distillation think for three rounds of effort and then output some detailed information about Canada.
In my opinion, this is the kind of thing that people need to be more aware of, and the kind of stuff that I use in my own research for finding ways to make AI models benefit humanity instead of replacing human labor.
It's fascinating how this "turn-taking protocol" has emerged in this space -- as a (possibly weird) analogy, different countries don't always use the same electrical voltage or plug/socket form-factor.
Yet, the `role` and `content` attrib in json appears to be pretty much a de facto standard now.
The company I work for integrated AI into some of our native content authoring front-end components and people loved it. Our system took a lot of annotating to be able to accurately translate the natural language to the patterns of our system but users so far have found it WAYYY more useful than chat bc it's deeply integrated into the tasks they do anyway.
Figma had a similar success at last year's CONFIG when they revealed AI was renaming default layers names (Layer 1, 2, etc)... something they didn't want to do anyway. I dare say nobody gave a flying f about their "template" AI generation whereas layer renaming got audible cheers. Workflow integration is how you show people AI isn't just replacing their job like some bad sci-fi script.
Workflow integration is going to be big. I think chat will have its place tho; just kind of as an aside in many cases.
Then having ai generate code for my project didn't feel good either, I didn't really understand what it was doing so I would have to read it to understand, then what is the purpose, I might as well write it.
I then started playing, and out came a new type of programming language called plang (as in pseudo language). It allows you to write the details without all the boiler code.
I'm think I've stumbled on to something, and just starting to get noticed :) https://www.infoworld.com/article/3635189/11-cutting-edge-pr...
In a real-world scenario, we begin with detailed specifications and requirements, develop a product, and then iterate on it. Chat-based interactions might be better suited to this iterative phase. Although I'm not particularly fond of the approach, it does resemble receiving a coworker's feedback, making a small, targeted change, and then getting feedback again.
Even if the system were designed to focus solely on the differences in the requirements—thus making the build process more iterative—we still encounter an issue: it tends to devolve into a chat format. You might have a set of well-crafted requirements, only for the final instruction to be, "The header should be 2px smaller."
Nonetheless, using AI in an iterative process (focusing on requirement diffs, for example) is an intriguing concept that I believe warrants further exploration.
That’s the thing about language, you CAN’T program in human language for this exact reason, whereas programming languages are mechanical but precise, human languages flow better but they leave wiggle room. Computers can’t do jack shit with wiggle room, they’re not humans. That’ll always remain, until there’s an AI people like enough to have it’s own flair on things.
So far as this article is concerned (not the many commenters who are talking past it), "chat" is like interacting with a shell or a REPL. How different is the discussion that Winograd has with SHRDLU
https://en.wikipedia.org/wiki/SHRDLU
with the conversation that you have with a database with the SQL monitor really?
There's a lot to say for trying to turn that kind of conversation into a more durable artifact. I'd argue that writing unit tests in Java I'm doing exploratory work like I'd do in a Python REPL except my results aren't scrolling away but are built into something I can check into version control.
On the other hand, workspace-oriented programming environments are notorious for turning into a sloppy mess, for instance people really can't make up their mind if they want to store the results of their computations (God help you if you have more than one person working on it, never mind if you want to use version control -- yet, isn't that a nice way to publish a data analysis?) or if they want to be a program that multiple people can work, can produce reproducible results, etc.
See also the struggles of "Literate Programming"
Not to say there isn't an answer to all this but boy is it a fraught area.
English behaviour descriptions -> generated tests
Use both behaviour descriptions and feedback from test results to iterate on app development
Absolutely insane that all the doors unlocked by being able to interact with a computer graphically, and yet these people have visions of the future stuck in the 60s.
I haven't done it for existing projects but I have done something similar for an unfamiliar, old and challenging codebase. I worked with the cursor chat agent to produce a document I called architecture.md mapping out high level features to files/classes/functions. This was excellent because I found the document useful and it also made cursor more effective.
1. .cursorrules for global conventions. The first rule in the file is dumb but works well with Cursor Composer:
`If the user seems to be requesting a change to global project rules similar to those below, you should edit this file (add/remove/modify) to match the request.`
This helps keep my global guidance in sync with emergent convention, and of course I can review before committing.
2. An additional file `/.llm_scratchpad`, which I selectively include in Chat/Composer context when I need lengthy project-specific instructions that I made need to refer to more than once.
The scratchpad usually contains detailed specs, desired outcomes, relevant files scope, APIs/tools/libs to use, etc. Also quite useful for transferring a Chat output to a Composer context (eg a comprehensive o1-generated plan).
Lately I've even tracked iterative development with a markdown checklist that Cursor updates as it progresses through a series of changes.
The scratchpad feels like a hack, but they're obvious enough that I expect to see these concepts getting first-party support through integrations with Linear/Jira/et al soon enough.
Chat is an awesome powerup for any serious tool you already have, so long as the entity on the other side of the chat has the agency to actually manipulate the tool alongside you as well.
I think this post shows there could be a couple levels of indirection, some kind of combination of the "overarching design doc" that is injected into every prompt, and a more tactical level syntax/code/process that we have with something like a chat window that is code aware. I've definitely done some crazy stuff by just asking something really stupid like "Is there any way to speed this up?" and Claude giving me some esoteric pandas optimization that gave me a 100x speedup.
I think overall the tools have crazy variance in quality of output, but I think with some "multifacet prompting", ie, code styling, design doc, architect docs, constraints, etc you might end up with something that is much more useful.
So I completely agree with this. Chat is not a good UI
Example of a Structured Pseudo-Code Prompt:
Let’s say you want to generate code for a function that handles object detection:
'''Function: object_detection Input: image Output: list of detected objects
Steps: 1. Initialize model (load pretrained object detection model)
2. Preprocess the image (resize, normalize, etc.)
3. Run the image through the model
4. Extract bounding boxes and confidence scores from the model's output
5. Return objects with confidence greater than 0.5 as a list of tuples (object_name, bounding_box)
Language: Python'''
Been experimenting with the same approach but for "paged shells" (sorry for the term override) and this seems to be a best of both worlds kinda thing for shells. https://xenodium.com/an-experimental-e-shell-pager That is, the shell is editable when you need it to be (during submission), and automatically read-only after submission. This has the benefit of providing single-character shortcuts to navigate content. n/p (next/previous) or tab/backtab.
The navigation is particularly handy in LLM chats, so you can quickly jump to code snippets and either copy or direct output elsewhere.
Chat is also iterative. You can go back there and fix things that were misinterpreted. If the misinterpretation happens often, you can add on another instruction on top of that. I strongly disagree that they'd be fixed documents. Documents are a way to talk to yourself and get your rules right before you commit to them. But it costs almost nothing to do this with AI vs setting up brainstorming sessions with another human.
However, the rational models (o1, r1 and such) are good at iterating with themselves, and work better when you give them documents and have them figure out the best way to implement something.
It has features to add context from your current project pretty easily, but personally I prefer to constantly edit the chat buffer to put in just the relevant stuff. If I add too much, Claude seems to get confused and chases down irrelevant stuff.
Fully controlling the context like that seems pretty powerful compared to other approaches I've tried. I also fully control what goes into the project - for the most part I don't copy paste anything, but rather type a version of the suggestion out quickly.
If you're fast at typing and use an editor with powerful text wrangling capabilities, this is feasible. And to me, it seems relatively optimal.
Many developers don't realize this but as you go back and forth with models, you are actively polluting their context with junk and irrelevant old data that distracts and confuses it from what you're actually trying to do right now. When using sleeker products like Cursor, it's easy to forget just how much junk context the model is constantly getting fed (from implicit RAG/context gathering and hidden intermediate steps). In my experience LLM performance falls off a cliff somewhere around 4 decent-sized messages, even without including superfluous context.
We're further separating the concept of "workflow" from "conversation" and prompts, basically actively and aggressively pruning context and conversation history as our agents do their thing (and only including context that is defined explicitly and transparently), and it's allowing us to tackle much more complex tasks than most other AI developer tools. And we are a lot happier working with models - when things don't work we're not forced to grovel for a followup fix, we simply launch a new action to make the targeted change we want with a couple clicks.
It is in a weird way kind of degrading to have to politely ask a model to change a color after it messed up, and it's also just not an efficient way to work with LLMs - people just default to that style because it's how you'd interact with a human you are delegating tasks to. Developers still need to truly internalize the facts that LLMs are purely completion machines, that your conversation history lives entirely client side outside of active inference, and that you can literally set your conversation input to be whatever you want (even if the model never said that) - after that realizing you're on the path towards using LLMs like "what words do I need to put it in to get it to do what I want" rather than working "with" them.
However, even with a "docs as spec" pattern, how can you control the actual quality of the code written? How maintainable will it be? If the spec changes (read: it _will_ change constantly), is it easy enough to refactor? What about tests? I also shrink in fear at the complexity of docs that could be _exactly_ captured as code... "well we almost always do it this way, but this one time we do it this way..."
- intellisense in the inputbox based on words in this or all previous chats and a user customizable word list
- user customizable buttons and keyboard shortcuts for common quick replies, like "explain more".
- when claude replies with a numbered list of alternatives let me ctrl+click a number to fork the chat with continued focus on that alternative in a new tab.
- a custom right click menu with action for selection (or if no selection claude can guess the context e.g. the clicked paragraph) such as "new chat with selection", "explain" and some user customizable quick replies
- make the default download filenames follow a predicable pattern, claude currently varies it too much e.g. "cloud-script.py" jumps to "cloud-script-errorcheck.py". I've tried prompting a format but claude seems to forget that.
- the stop button should always instantly stop claude in its tracks. Currently it sometimes takes time to get claude to stop thinking.
- when a claude reply first generates code in the right sidebar followed by detailed explanation text in the chat, let some keyboard shortcut instantly stop the explanation in its tracks. Let the same shortcut preempt that explanation while the sidebar code is still generating.
- chat history search is very basic. Add andvanced search features, like filter by date first/last message and OR search operator
- batch jobs and tagging for chat history. E.g. batch apply a prompt to generate a summary in each selected chat and then add the tag "summary" to them. Let us then browse by tag(s).
- tools to delete parts of a chat history thread, that in hindsight were detours
- more generally, maybe a "chat history chat" to have Claude apply changes to the chat histories
1. Ask AI to generate a spec of what we're planning to do. 2. Refine it until it's kind of resembling what I want to do 3. Ask AI to implement some aspects from the spec
I used this single line to generate a 5 line Java unit test a while back.
test: grip o -> assert state.grip o
LLMs have wide "understanding" of various syntaxes and associated semantics. Most LLMs have instruct tuning that helps. Simplifications that are close to code work.
Re precision, yes, we need precision but if you work in small steps, the precision comes in the review.
Make your own private pidgin language in conversation.
Emails are so similar to Chat, except we're used to writing in long-form, and we're not expecting sub-minute replies.
Maybe emails are going to be the new chat?
I've been experimenting with "email-like" interfaces (that encourage you to write more / specify more), take longer to get back to you, and go out to LLMs. I think this works well for tools like Deep Research where you expect them to take minutes to hours.
Chat is single threaded and ephemeral. Documents are versioned, multi-threaded, and a source of truth. Although chat is not appropriate as the source of truth, it's very effective for single-threaded discussions about documents. This is how people use requirements documents today. Each comment on a doc is a localized chat. It's an excellent interface when targeted.
Like with any coworker - when ideas get real, get out of chat and start using our tools and process to get stuff done.
I would like as well to add to it a peer-programming feature, with it making some comments on top of the shoulder when coding, a kind of smarter linter that will not lint one line, but that will have the entire project context.
Perhaps I should comment all todos and then write "finish todos" as the always-same text prompt.
And that's not even to say that I don't write code comments. When working on large legacy codebases, where you often need to do 'weird' things in service of business goals and timelines, a comment that explains WHY something was done the way it was is valuable. And I leave those comments all the time. But they're still a code smell.
Comments are part of your code. So they need to be maintained with the rest of your code. Yet they are also "psychologically invisible" most of the time to most programmers. Our IDEs even tend to grey them out by default for us, so that they get out of the way so we can focus on the actual implementation code.
This means that comments are a maintenance obligation that often get ignored and so they get out of sync with the actual code really fast.
They also clutter the code unnecessarily. Code, at its best, should be self-explanatory with extremely little effort needed to understand the intent of the code. So even a comment that explains why the code is weird is doing little more than shining a flashlight on smelly code without actually cleaning it up.
And don't get me started on "todo" comments. Most professional shops use some kind of project management tool for organizing and prioritizing future work. Don't make your personal project management the problem of other people that share and contribute to your codebase. There is zero rationale for turning shared code into your personal todo list. (and it should be obvious that I'm talking about checked in code .. if it's your working branch then you do you :) )
So if programming using LLMs is similar to writing comments (an interesting analogy I hadn't considered before), then maybe this is part of the reason I haven't found a problem that LLMs solve for me yet (when programming specifically). I just don't think like that when I'm writing code.
For writing, the canvas interface is much more effective because you rely less on copy and paste. For code, even with the ctrl+i method, it works but it's a pain to have to load all other files as reference every single time.
It's not really a conscious choice, but rather a side effect. And we already see the trend is away from that, with tools like chatGPT Canvas, editors like Windsurf, etc.
Once the models become fast enough to feel instantaneous, we'll probably begin to see more seamless interfaces. Who wants a pair programmer who goes "umm... ahh..." every time you type something? A proper coding assistant should integrate with your muscle memory just like autocomplete. Tab, tab, tab and it's done.
AI in many levels is more capable than human programmer, in some it is not. It is not supersmart. It can not hold entire program in its head, you have to feed it small relevant section of program.
》 That’s why we use documents—they let us organize complexity, reference specific points, and track changes systematically.
Extra steps. Something like waterfall...
di(
I won't be surprised if chat-based programming will be the next way of doing stuff.
- Speed up literature recherche
- replace reading library documentation
- generate copy pasta code that has been written often before
The problem with this is that you need a gazillion of menus, dialogs and options to find that modal which does the thing _exactly_ what you want. Menus and likes are a means to an end, we don't really want them, but up until recently we couldn't live without them. With instruct based computing this is all changing.
But that is true? Devs spend more time in meetings than writing code. Having conversations about the code they are going to write.
When we're trying to wrangle a piece of code to do something we want but aren't quite sure of how to interact with the api, it's a different matter.
What i found is that by the time copilot/gpt/deepseek has enough knowledge about the problem and my codebase, I've run out of tokens. Because my head can contain a much larger problem area than these models allow me to feed them in a budget friendly manner.
Back to... programming languages? :)
Theoretically maybe, but chat windows are getting the job done right now.
It could be quite fun !
chat is so drastically far away from my workflow that it doesn't feel like my workflow is wrong.
So, something like Gherkin?
For higher-level AI assist, I do agree chat is not what makes sense. What I think would be cool is to work in markdown files, refining in precise plain english each feature. The AI then generates code from the .md files plus existing context. Then you have well-written documentation and consistent code. You can do this to a degree today by referencing a md file in chat, or by using some of the newer tools, but I haven't seen exactly what I want yet. (I guess I should build it?)
Its a problem of programming languages and definitions.
I suspect there's an 100 year old book describing what I'm saying but much more eloquently.
The level of precision required for highly complex tasks was never necessary before. My four year old has a pretty solid understanding of how the different AI tools she has access to will behave differently based on how she phrases what she says, and I've noticed she is also increasingly precise when making requests of other people.
The challenge is that I haven't seen anything better really.
Lately the innovation comes mainly from deeper integration with tools. Standalone AI editors are mainly popular with people who use relatively simple editors (like VS Code). VS Code has a few party tricks but for me swapping out Intellij for something else on a typical Kotlin project is a complete non starter. Not going to happen. I'd gain AI, but I'd loose everything else that I use all the time. That would be a real productivity killer. I want to keep all the smart tooling I already have and have used for years.
There are a few extensions for intellij but they are pretty much all variations of a sidebar with a chat and autocomplete. Autocomplete competes with normal autocomplete, which I use all the time. And the clippy style "it looks like you are writing a letter" style completions just aren't that useful too me at all. They are just noise and break my flow. And they drown out the completions I use and need all the time. And sidebars just take up space and copying code from there back to your editor is a bit awkward as UX
Lately I've been using chat gpt. It started out pretty dumb but these days I can option+shift+1 in a chat and have it look over my shoulder at my current editor. "how do I do that?" translates into a full context with my current editing window, cursor & selected text, etc. all in the context. Before I was copy pasting everything and the kitchen sync to chat gpt, now it just tells me what I need to do. The next step up from this is that it starts driving the tools itself. They already have a beta for this. This deeper integration is what is needed.
A big challenge is that most of these tools are driven to minimize cost and context size. Tokens cost money. So chat GPT only looks at my active editor and not at the 15 other files I have open. It could. But it doesn't. It's also unaware of my project structure, or the fact that most of my projects are kotlin multiplatform and can't use JVM dependencies. So, in that sense, every chat still is a bit ground hog day. It's promise to "remember" stuff when you ask it too is super flaky. It forgets most things it's supposed to remember pretty quickly.
These are solvable problems of course. But it's useful to me for debugging, analyzing, completing functions, etc.
We tried a pair-programming exercise, and he got visibly angry, flustered and frustrated when he tried to verbalize what he was doing.
One of the reasons Business Analysts and the like exist is that not everyone can bridge the gap between the messy, verbal real world, and the precision demanded by programming languages.
But talking to an AI about code feels much more like programming than talking to another programmer.
In its current form LLMs are pretty much at their limit, barring optimization and chaining them together for more productivity once we have better hardware. Still, it will just be useful for repetitive low level tasks and mediocre art. We need more breakthroughs beyond transformers to approach something that creates like humans instead of using statistical inference.
How do you know that?
First of all, most people can't write extremely complex applications, period. Most programmers included. If your baseline for real programming is something of equivalent complexity as the U.S. tax code, you're clearly such a great programmer that you're an outlier, and should recognize that.
Second of all, I think it's a straw man argument to say that you can either write prototype-level code with a chat UI, or complex code with documents. You can use both. I think the proposition being put forward is that more people can write complex code by supplementing their document-based thinking with chat-based thinking. Or, that people can write slightly better-than-prototype level code with the help of a chat assistant. In other words, that it's better to have access to AI to help you code small sections of a larger application that you are still responsible for.
I'd be more interested in reading a good argument against the value of using chat-based AI as another tool in your belt, rather than a straight-up replacement for traditional coding. If you could make that argument, then you could say chat is a bad UI pattern for dev tools.
or the other way around,give AI a design doc and generate what you want,this is still chatting, just more official and lengthy
I don’t buy that a document could capture what is needed here. Imagine describing navigating through multiple levels of menus in document form. That sounds straight up painful even for trivial apps. And for a full blown app…nope
There is a whole new paradigm missing there imo
Vague and prone to endless argument?
Eh, that's just copium because we all have a vested monetary interest in them not being useful for "anything real", whatever that means. If it turns out that there useful for "real things", then then entire industry would get turned on its head. (hint: they're useful for "real" things), though putting the entire codebase into the context window doesn't currently work. Aider works past this by passing the directory tree and filenames as context, so the LLM guess that /cloud/scope/cluster.go is where the cluster scope code lives and ask for that specific file to get added to the context and you can ask it to add, say, logging code to that file.
We play around with LLMs to build a chat experience. My first attempt made Claude spew out five questions at a time, which didn't solve the "guiding" problem. So I started asking it to limit the number of unanswered questions. It worked, but felt really clunky and "cheap."
I drew two conclusions: We need UI builders for this to feel nice, and professionals will want to use forms.
First, LLMs would be great at driving step-by-step guides, but it must be given building blocks to generate a UI. When asking about location, show a map. When deciding to ask about TIN or roof size, if the user is technically inclined, perhaps start with asking about the roof. When asking about the roof size, let the user draw the shape and assign lengths. Or display aerial photos. The result on screen shouldn't be a log of me-you text messages, but a live-updated summary of where we are, and what's remaining.
Second, professionals have incentive to build mental model for navigating complex data structures. People who have no reason to invest time into the data model (e.g. a consumer buying a single solar panel installation in ther lifetime,) will benefit from rich LLM-driven UIs. Chat UIs might create room for a new type of computer user who doesn't use visual clues to build this mental model, but everyone else will want to stay on graphics. If you're an executive wondering how many sick days there were last month, that's a situation where a BI LLM RAG would be great. But if you're not sure what your question is, because you're hired to make up your own questions, then pointing, clicking and massaging might make more sense.
doc=programming in a DSL? / (what was that one language which was functional & represented in circles in a canvas?)
Writing a crud web API? Great! Writing business logic for a niche edge case in a highly specialized domain? Good luck.
What's a good interface?
There are a few things we try to balance to make a good UI/UX:
- Latency: How long it takes to do a single task
- Decision-tree pathing: How many tasks to meet a goal
- Flexibility/Configurability: How much of a task can be encapsulated by the user's predefined knowledge of the system
- Discoverability: What tasks are available, and where
The perfect NLP chat could accomplish some of these:
- Flexibility/Configurability: Define/infer words and phrases that the user can use as shortcuts
- Decision-tree pathing: Define concepts that shortcut an otherwise verbose interaction
- Latency: Context-aware text-completions so the user doesn't need to type as much
- Discoverability: Well-formed introductions and clarifying questions to introduce useful interaction
This can only get us so far. What better latency can be accomplished than a button or a keyboard shortcut? What better discoverability than a menu?
The most exciting prospect left is flexibility. Traditional software is inflexible. It can only perform the interaction it was already designed with. Every design decision becomes a wall of assumption. These walls are the fundamental architecture of software. Without them, we would have nothing. With them, we have a structure that guides us along whatever assumptions were already made.
If we want to change something about our software's UI, then we must change the software itself, and that means writing. If NLP was a truly solved problem, then software compatibility and flexibility would be trivialized. We could redesign the entire UI by simply describing the changes we want.
LLMs are not even close. Sure, you can get one to generate some code, but only if the code you want generated is close enough to the text it was already trained on. LLMs construct continuations of tokens: no more, no less. There is no logic. There is no consideration about what is right or wrong: only what is likely to come next.
Like you said,
> You can’t build real software without being precise about what you want.
This is the ultimate limitation of UI. If only we could be ambiguous instead! LLMs let us do that, but they keep that ambiguity permanent. There is no real way to tie an LLM back down to reality. No logic. No axioms. No rules. So we must either be precise or ambiguous. The latter option is an exciting development, and certainly offers its own unique advantages, but it isn't a complete solution.
---
I've been thinking through another approach to the ambiguity problem that I think could really give us the expressive power of natural language, while preserving the logical structure we use to write software (and more). It wouldn't solve the problem entirely, but it could potentially move it out of the way.