I would be very interested to know from everyone here:
What is the most impressive thing you have managed to get an AI to code - WITHOUT having to babysit it or give it any tips hints or corrections that a non-coder would have been unable to do.
What is the most impressive thing you have managed to get an AI to code - WITHOUT having to babysit it or give it any tips hints or corrections that a non-coder would have been unable to do.
Case one. I configured IPsec VPN on a host machine which run docker containers. Everything worked from the host itself, however containers were not able to reach IPsec subnet. I spent quite a bit of time, untangling docker iptables rules, figuring out how iptables interacts with IPsec, running tcpdumps everywhere. However my skills were not enough, probably I would resolved the issue given more time, however I decided to try ChatGPT. I made a very thorough question, added everything I tried, related logs and stuff. Actually I wanted to ask the question on some Linux forums, so I was preparing the question. ChatGPT thought few minutes and then spewed one iptables command which just resolved the issue. I was truly impressed.
Case two. I was writing firmware for some device using C. One module was particularly complex, involved management of two RAM buffers and one external SPI buffer. I spent two weeks writing this module and then asked ChatGPT to review my code for major bugs and issues. ChatGPT was able to find out that I used SPI to talk to FRAM chip, it understood that my command usage was subtly wrong (I sent WREN and WRITE commands in the one SPI transaction) and highlighted this issue. I tried other modes, I also tried Claude, but so far only o1 pro was able to find that issue. This was impressive because it required to truly understand the workflow of the code and it required extensive knowledge of protocols and their typical usages.
Other than that, I don't think I was impressed by AI. Of course I'm generally impressed by its progress, it's marvellous that AI exists at all and can write some code that makes sense. But so far I didn't fully integrate AI into my workflows. I'm using AI as Google replacement for some queries, I use AI as a code reviewer and I'm using Copilot plugin as a glorified autocomplete. I don't generate any complex code with it and I rarely generate any meaningful code at all.
I wonder over time how those small, infrequent updates might hamper the ability to perform the next, small infrequent update (as your code begins to resemble less and less any examples the AI might have seen and more and more a kludge of differing styles, libraries, etc.), but that's really not any different than how most projects like a personal website operate today.
Here’s the initial conversation if you’re interested: https://grok.com/share/bGVnYWN5_9ce1bed4-7136-4761-b45e-0ab0...
I think this already falls out of OP's guidelines, which you pointed out are quite strict. They also happen to be the guidelines an AI would need to meet to "replace" competent engineers.
I haven't heard of any prediction that AI will completely replace all programmers.
Programmers aren't paid to generate code. They're paid to figure out how to do X Y and Z business initiatives. Unless you don't have that many initiatives, how much sense does it make to let go of the people who can do that for you? It makes sense when companies are trying to cut costs, but that happens when the cost of borrowing is higher than the realizable profit margin
Though I’ll admit LLMs have been weirdly good at regex.
I find that question impossible to answer, because my programming experience influences everything I use LLMs for. I can't turn that part of my brain off.
Getting AI to write code for you starts with understanding what's possible, and that's hugely informed by existing programming knowledge.
I won't ask an LLM to build me something unless I'm reasonably confident it will be able to do it - and that confidence comes from 25+ years of programming experience combined with 2+ years of intuition as to what LLMs themselves can handle.
Essentially, for "real-life work scenarios", the performance is not that great comparing to gpt-3.5 with the exception of Claude 3.7 and GPT-4.5. Bear in mind, the question was not "challenging" in a "genius thinking required" way.
That being said, I use LLM regularly for discovery. They do waste some of your time because of hallucinations but you can call their bullshit most of the time and if not, it is still faster than Googling.
Make a vbscript to toggle scroll-lock every 15 seconds ... to prevent system from auto locking
Ad hoc text processing ... like strip the HTML from this snippet (drop-down list copied from a web page DOM)
Did this over a few weeks in my free time and now it has all of the features of accessibility monitoring SaaS that I was previously paying $600/mo for.
It doesn't always work, but I have an easy, quick way to test it out. When it does work, I've often saved lots of time.
So like, I have code to find optimal production chains and solve the node graph, using ILP through pyomo or Z3.
Some of the optimal production chains are ... a lot of nodes, as are plenty of factories. Without really good layout, it becomes a mess. Existing modelers sort of suck and have no auto-layout.
I bridged Elk (and elk.js, actually) into python, but wanted a fallback since this was ... crazy enough already and who knows if it will work on anyone else's computer :)
So AI wrote me about 4000 lines of just about 100% correct python graph layout code. In batches, one algorithm at a time, that i then combined. I did have to tell it what I wanted piece by piece, so i had to learn a lot about it - i could not get it to do all the pieces at once iteslf. I suspect due to context length limitations, etc.
It comes very close to ELK when it comes to this type of layout (elk supports other layouts, edge routing, etc), and implements the same algorithms with the same advanced techniques.
And TIL about elk! I've had kind of a half project, called `gstd`, which tries to create a standard library/API to make working with node graphs in code as easy as working with other abstract data types, like Arrays. One of the important bits is that it lets you console log a graph so you can see it. I've been using d3 force directed graphs for layouts, and have played around with some custom layout algorithms, but have yet to find a good solution. Elk might actually be just what I've been looking for there!
Thanks for taking the time to respond!