Ask HN: How do you use AI for software engineering? | Hacker News Reader

Ask HN: How do you use AI for software engineering?

Hey HN,

It feels like so much has changed in the last 18 months and I feel like I'm behind the curve.

I love software engineering and have been somewhat hesitant to use these new tools. But I am now committed to becoming an expert on using AI in my development workflow.

What are the tips you would give to somebody who is trying to figure this stuff out for the first time?

I have Copilot installed. Are there other tools that are making a big difference? Any particular workflows that make these tools most effective? Etc. I'm open to hearing everything!

Thank you all!

EDIT: So many amazing answers. One thing I'm curious about... if you can, I would love to see example transcripts from ChatGPT. Obviously many won't be able to be shared, but hopefully somebody has something that can!

View on HN

In my opinion, using these tools daily and extensively, AI and AI copilots are an extreme force multiplier for in writing boilerplate code, debugging simple (but non-obvious) mistakes in existing code, and context immersion ("How do I X in framework Y"). It truly helps elevate the craft of SWE above writing code to complexity management and system evolution.

I also feel like this generation of senior/staff SWEs (the senior pre-AI engineer) is in a very unique position for the following reason: we know what the correct code should look like, because we've written it ourselves many times, and we know how to review and modify it quickly. I don't think new SWEs will have that same ability. At the same time, that ability may not be required in the future as AI gets better and better at following instructions correctly.

AI's future in coding is uncertain. I think there is a new unrealized engineering platform on the horizon that can only be realized through a very small and focused group of architects using AI (maybe even 1 person), and that this new platform, which is focused on complexity management and system evolution, will change how we "code" completely, at the same time resetting AI's copilot utility in this new world.

This is exactly how I'm seeing it right now. Those bots are generally still wrong in some ways, but I can generally spot it in seconds and rephrase those parts after a bit of back and forth. That is still a bit annoying currently, because they end up re-printing the entirety of long messages for singular changes, and they're not so good at changing subtle elements reliably despite talking with absolute certainty.

I'm currently dealing with immense legacy tooling (90s to 2010s) and codebases (C, Cpp, C#, Python, JS) in a central team that fell apart a year before I joined. They had 15+ people with diverse profiles who made snippets and undocumented solutions everywhere, and pretty much all quit at the same time due to working conditions at that time. Needless to say it's a daunting task to try and understand everything, even more so when you discover it all as it breaks down and prevents 500 employees from using a functionality you didn't even know existed.

While I can survive and navigate just fine the situation, it's been very helpful to use LLMs to run a parallel effort that helps me understand just why the hell somebody wrote a self-altering stored procedure in a 800GB MsSQL DB which fails due to poor design and high recursion. Sometimes it is just about fixing some syntax, but often enough it is about managing expectations and offering replacement solutions which create less technical debt - with minimal interruption to the entire company's workflow.

It isn't abnormal for my day to require reverse engineering and fixing a 500-line stored procedure, a Go CLI, multi-CI imbricated pipelines, IIS websites, cronjobs, and even c-shell scripts (!). That's when I'll occasionally throw a big pile into chatgpt as a bit of a glorified rubber ducky. Also tried local LLMs but my 3070ti can't sustain any worthwhile model.

Many of us here are in a very unique position of having seen the entire stack - from quantum tunneling all the way up to your CI badly parsing a groovy script or why an endlessly long JS program is stumbling over itself because its writer didn't factor in any estimate of runtime complexity. For me this is a beautiful and lucky thing to see through all those orders of magnitude with a certain equanimity, and I kinda wish I could find it in my own colleagues as much as I'm already seeing it in the illusion of competency created by LLMs.

You helped me find a good analogy of what a LLM really is, basically an improved rubber-duck that can guide you.

> I also feel like this generation of senior/staff SWEs (the senior pre-AI engineer) is in a very unique position for the following reason

Yeah, it'll be interesting to see how that changes things.

I think even pre-AI there's already a big difference in old-school programmers and the current generations of graduates.

People like myself who grew up programming in the 8-bit era are used to having to design things from scratch, out of necessity, whether that's custom algorithms or data structures, or something like a parser or editor, or whatever. It seems that current generations are more used to assembling things out of lego blocks - more system integration than what I would consider as programming. I guess this is fine when it works - when you can find the pieces you need, but if that's the only skill you have then there is nothing to fall back on if you need to create something complex truly from scratch.

100% agree here !

On one hand, it makes me really think about my first job - when we automated a lot of things, but the people who did it knew how things really worked. After 2 years, all the newcomers only knew if the indicator is red or green without any understanding of what's really going in - it went downhill from here.

On the other hand, if I understand your last paragraph correctly, the real question is do we really need to know all those details ? Would it just be better to have it automated, and the AI agent just ask you questions relating to the desired outcome and outlines the limitations and impacts of your choices, as well as the options available.

My interest is massively piqued by the last bit... any rough ideas of what that looks like?

I'm not sure, but I think of a gesture and voice commanded coding, that is AI guided, and beautifully immersive VR presented and edited, where AI guided is a variable of unknown limit describing how intelligent it could get. I reckon Mcluhan would call it a different medium, in the way he does reading press printing vs a medieval manuscript. Whatever it is, it'll be significant.

Totally second this

Also for refactoring/creating tests and writing/fixing docs

About the second point, yes. Sometimes the LLM will create "almost correct" code but you definitely need to understand it and know why it is wrong.

I use ChatGPT (paid GPT-4) as a quick & competent secretary. Mostly for discussion of architecture, bouncing ideas and seeing if it can spot any problems, and for questions answerable by reading manuals. Stuff that previously meant a lot of manual searching by myself is now a quick question.

For code, I use it rarely. Mostly for small snippets, like shell scripts or SQL queries. It makes too many mistakes when dealing with bigger problem spaces and validating its work takes so much effort that I might as well write the code myself.

Overall I think it has a significant positive impact on saving me time, even though I still write almost all of the code myself. That's because writing code has never been a significant time chunk of my work. Figuring out what exactly to write is the hard part. Once you can describe it in detail, the actual mechanical act of converting it from English to a programming language is rather trivial.

I find the paid version of ChatGPT to be an effective pair programmer with me. From ideas to initial scaffolding to get my thinking straight about outcomes. I write mostly Python, and I find it great at typing my functions and writing docstrings. I also write alot of infrastructure code for the AWS Cloud, and I find ChatGPT better at that than Code Whisperer/Q, but I expect in the AWS context it will catch up.

This is what I do. It got much better once I upgraded to the paid GPT4 access. I've been playing around with Claude3, I have a mild preference for the GPT4 but it's close.

I use Kagi Ultimate for access to GPT 3.5, 4, 4 Turbo, Claude 1, 2 3, and Mistral Small and Large. I also run a local language model that I can pass data to that my employer wouldn't be happy with me shipping to external APIs.

I use the language models conversationally as a supplement to, and interface to documentation. Things like high level design questions. I also use it to parse through error logs that I don't understand. Sometimes it can point me in the right direction faster than doing web searches for random snippets in the log. My actual code generation is fairly minimal. At most I usually just ask it for an example to convey an idea in some language, and then I adapt the example to meet my needs.

1) Glorified doc search

2) Mechanical operations (e.g. translate this SQL query into a knex query)

3) Code review, verification of modern idiomatic approaches in languages I use infrequently

Very interesting. Do you use ChatGPT for these?

I'd mostly been using ChatGPT until recently. Now I've mostly been using Gemini 1.0 Ultra.

I personally have found that ChatGPT is great to bounce ideas and concepts off of. If I’m unfamiliar with something or I’m trying to learn a new skill or get a second opinion on some design or architecture details, it’s very helpful in that regard.

When it comes to pure code, ChatGPT will deceive you. I’m at the point now where I’m very hesitant to copy and paste ChatGPT code. I’ll have to completely vet it first, or it’ll have to literally just be a line or two. And even then I’ll likely have to refactor it. ChatGPT is just wrong too often when it comes to the actual code either due to lack of context about the problem or just downright hallucinations. It’s even gotten some boilerplate wrong.

When it comes to Copilot, I honestly haven’t been able to quantify how beneficial it’s been for me. I’ve often had to delete or modify its code. I think it could partially be because I’m not as skilled with it. It could also be that I’ve just had to write a lot of JS lately, and there’s probably a ton of bad JS that it’s been trained on. I remember being more impressed with it when I write Rust. But I’m also new with Rust. I need to try it with something like C#, which has more difficultly than JS but I’m also proficient with it.

All in all, AI tools for me have been just that, tools I can use to help me get the job done.

In all probability it's deceiving you the rest of the time as well. Based on what you've described here, you're trusting its expertise when you don't have much domain expertise of your own, but find it not to be particularly expert at all on matters where you yourself have suffient context and expertise to know better.

This is a common phenomenon with humans, FWIW. The same sort of thing happens with traditional information sources. For example, when a media outlet reports on things we don't know much about personally, we believe them. Then they cover something where we have direct domain expertise and find all manner of misunderstandings and errors trivially, but instead of suspecting that they're likely just as wrong about a lot of other things as well, we assume it's a special case where they just got our special knowledge domain wrong.

In any case, to answer the OP... I use these new AI tools to generate content where the details don't matter and the cost of being wrong is near zero. Such as, graphics for slides, market/product/pitch blurb pablum, etc.

I use them to compensate for my limited artistic/graphic design skills and to overcome my propensity to tediously labor over copy despite that copy being basically throwaway.

This is indeed well-known, in recent years I've seen it referred to as Gell-Mann Amnesia: https://www.epsilontheory.com/gell-mann-amnesia/

If I had to really try and pinpoint where my discomfort for this latest hypecycle is coming from, it's probably that I have this sense that the dynamic behind Gell-Mann Amnesia is being cynically exploited by the interests overselling these new products.

Kind of in the same vein as the strategies around gamification, where some identified frailty of the human psyche is being leaned on for cash.

It's not quite a con because the technology is useful and certainly worth something, but it's not non-exploitive either.

To me, the biggest value of Copilot is that you get unlimited access to a gpt-4 chat for half the price of ChatGPT Plus via Copilot Chat.

The context sensitive stuff like highlighting a line of code and asking questions about it and the completions are okay too but the chat is the most useful part imo.

Yeah, great for architecture and design especially in areas I feel impostor syndrome from knowing just enough to be dangerous. I'd say it extends the horizon of possibilities, especially in filling in grey areas.

This all makes sense, thank you. And with ChatGPT are you using the paid version? I have been experimenting with both and found the free version to be very bad. The paid version seems to make fewer mistakes.

I've started to look at the quick answers more in Kagi which use AI. That's saved me some time.

I have Copilot installed, but don't really use it much. I had to disable the auto-complete, because it uses tab to accept, which made it impossible to simply use the tab key... you know, to indent my code. The idea that I would need to hit escape every time I wanted to press tab was stupid. Accepting a suggestion should require a modifier key, and there didn't seem to be a setting to change this. Therefore, I only use Copilot when I explicitly invoke it, which is almost never. (If someone knows how to make this better, I'm all ears)

This, along with a string of bad suggestions is what made me back away from Copilot as well.

I stumbled across statistics my company is looking at to judge the value of Copilot. It looks like on average, 20% of suggestions are accepted. So 80% of the time it’s completely wrong. From my limited experience, I can only assume the 20% are very minor things.

I'm not a dev any more so I guess saying I use AI for software engineering is a stretch, but I run Mistral 7b locally and use it to assist with engineering management things. Recently I took a several lists of things from the 6 teams I run, that were all roughly the same things but written differently, and prompted Mistral to dedupe the lists based on the semantics of what my leads had written. It didn't do a terrible job. To have done the same thing by hand would have taken a couple of hours. I was impressed.

I’ve been doing a lot of AWS infrastructure setup lately and use ChatGPT frequently to generate possible solutions and starting examples in CDK. It doesn’t solve the problems reliably by any means, but it’s good for getting an idea of an approach so I can go read the docs. It can even output a diagram if you ask for mermaid format.

It’s good for “rubber duck” debugging when things don’t work. I just give a brief description of what I’m trying to do, and paste in some code with an error message or some log entries, and it’s pretty great at pointing to the problem and suggesting a fix.

Since its capabilities are opaque, I constantly try new things. Yesterday I asked it “what was the earliest version of sass-loader that supported the quietDeps option” and it actually answered the question, with a citation. That saved a lot of time!

There was a cookie error that showed up as a pop up in the Chrome devtools. I couldn’t copy and paste the error from the pop-up, but I took a screenshot of it and pasted that into a chat. ChatGPT read it and gave me a little tutorial on the SameSite cookie setting.

It seems to be getting a little bit more introspective lately. Last week I asked it about something, and it described a solution, then apologized and said that wouldn’t actually work, and gave a different answer. I mean all of that happened in one answer, not because I prompted it. Then I asked for clarification, and it apologized again and said the original solution would work. (It wouldn’t).

ChatGPT (paid) for everything from mentoring to memory-refreshing to creating wiki pages. It's a complete game changer for me.

I also use copilot/chat in VSCode and have used GitHub copilot in the terminal in the past.

But really nothing above is as useful as ChatGPT.

Apparently Claude 3 Opus is it's first serious competitor, I haven't tried it.

Ignore negative feedback on ChatGPT from anybody who hasn't used the paid version, free stuff isn't the same.

I recently upgraded to the paid version of ChatGPT and it's actually proven to be a pretty effective partner in getting unstuck.

The two areas where it's been particularly helpful:

- Tricky Postgres queries - it essentially explains the pretty dry docs to you like you're five

- Partially because I use nvim without intellisense (made the switch after 23 years of vim last year!) - occasional TypeScript type issues

I'm using Jetbrains AI, ChatGPT 4 and Cursor.

Cursor is great for diving into third party codebases as it can read your whole project if you tell it to. So, let's say you're in some frontend file and don't understand why it behaves weird, it can point you to a different file (e.g. "Hey, there's a global CSS file that has an impact here!"). I hope Jetbrains will catch up, soon.

Jetbrains mainly differentiates from ChatGPT through their IDE integration. You can select code and ask it to refactor, find issues or chat about it. It proposes Git commit messages.

When I feel lazy, I use AI to create simple functions, e.g. "write a function that recursively reads all files in a given directory and returns all file names containing a given string". But most of the time it's more about dealing with existing code than implementing something new.

I often use ChatGPT to generate code for specific tasks. For example, maybe I need to pull thousands of files from S3, perform some sort of manipulation, and then send them back to S3. I can ask ChatGPT about this and have it generate the basic functionality of what I want, but then refine it myself to get the details exactly how I want them. So I find it extremely good at getting me 80%+ through basic tasks, saving me often many minutes on the mundane parts of these tasks. I gladly pay the $20/month for this.

I have briefly tried it in my IDE, but so for I don’t like the experience of any of the plugins. They feel a bit awkward to me in their current state. But, I know other devs who love the integration directly in their IDE.

Mostly ChatGPT but I recently added https://aider.chat/ semi successfully into my workflow. It doesn't replace ChatGPT but it has replaced certain tasks that involve editing files. It uses GPT-4 (or Claude Opus) to write and apply diffs to my git repo and automatically includes my code in the context. In summary:

Design - ChatGPT

Major features - Aider

Fixups - GitHub Copilot

I'm thinking about forking Aider because I think it could easily handle the design portion if I had more control of how it applied diffs. Right now it decides when it applies diffs in chat and sometimes you want to go back and forth before it starts to wreck a file without enough requirements.

If you have time, could you write a bit more about how you use Aider?

Anytime I find myself coding something easy, but repetitive and boring, I hand it off to ChatGPT.

“I’m going to give you a list of 50 variables and I want you to format them into functions like this:”

It’s faster and easier than writing a script to do it.

That's an interesting use case - a bit like editor macros on steroids.

I write frontend web code in Rust (dominator). There is no plugin for Figma that handles this out of the box, but by copy pasting the CSS and putting it in a comment, Co-Pilot can then infer the transformation to Rust syntax. In the past I would have taken the time to write my own parser/mapper, now I don't have to, nor does _anyone_, it's even _better_ for it to learn from local context e.g. for my own personal formatting style etc.

Are there any tutorials you recommend to see how to better leverage AI? I looked at some videos on YouTube and it seemed that the overall workflow was quite slow.

Same as pretty much everyone else, give copilot the easy/dumb stuff. I sometimes throw it oddball questions I know it won’t give a great answer to, but an ok answer is sometimes enough information for further targeted searching of my own. I started work on a side project recently and was missing copilot enough (only had it on the work computer) to go buy a personal license.

I like re-implementing libraries for learning. Take something you know very well and redo it with something you are trying to learn.

For example, making both redux and react-query drop in replacements using just rxjs.

I learned a great deal from these. I'm not going to use them but now I understand rxjs way more

I sometimes have nasty bugs in my code which are not directly visible to my eyes, and I ask ChatGPT 4 to find the bugs, and it usually does, and explains what I'm doing wrong.

In some sense, it's a bit like pair programming... with a partner available 7/24 and responding immediately.

The biggest gain for me has been avoiding Google/SO for API references and "How do I just do this thing" type questions. The productivity increase is insane versus sifting through the nightmare of whatever web results have become today.

Someday imagine we save only prompts in git & generate entire app in the build pipeline. This expects app generation is always consistent.

However, not using ai today. Mainly because, I am running legacy ucode.

I don't use it at all. The handful of times I have tried to ask ChatGPT questions or to implement some algorithm it has made up or gotten wrong integral parts.

It takes a bit of practice and a few incantations to use it effectively.

I use it as conversational search. That said, I'm actually building a platform for writing software (and more) with AI.

phind.com has pretty much replaced googling for me, I've been also using it a lot for code reviews.

As a better search. Also Can be like junior dev to hand off boilerplate or easy tasks.

The usual:

ChatGPT to find fixes for weird errors.

Copilot to autocomplete code.

I feel guilty for not using paid ChatGPT subscription more.

ChatGPT to generate what amounts to building blocks and then assemble them by hand.

Reply on news.ycombinator.com