I'm not great at remembering specific quirks/pitfalls about secondary languages like e.g. what the specific quoting incantations are to write conditionals in Bash, so I rarely wrote bash scripts for automation in the past. Basically only if that was a common enough task to be worth the effort. Same for processing JSON with jq, or parsing with AWK.
Now with LLMs, I'm creating a lot more bash scripts, and it has gotten so easy that I'll do it for process-documentation more often. E.g. what previously was a more static step-by-step README with instructions is now accompanied with an interactive bash script that takes user input.
Bash scripts are essentially automating what you could do at the command line with utility programs, pipes, redirects, filters, and conditionals.
If you're getting very far outside of that scope, bash is probably the wrong tool (though it can be bent to do just about anything if one is determined enough).
As it happens, I think co-pilot is a pretty poor user experience, because it's essentially just an autocomplete, which doesn't really help me all that much, and often gets in my way. I like using Cursor with the autocomplete turned off. It gives you the option to highlight a bit of text and either refactor it with a prompt, or ask a question about it in a side chat window. That puts me in the driver seat, so to speak, so I (the user) can reach out to AI when I want to.
I have seen mostly senior programmers argue why ai tools don't work. Juniors just use them without prejudice.
The trouble is that they seem to be getting worse. Some time ago I was able to write an entire small application by simply providing some guidance around function names and data structures, with an LLM filling in all of the rest of the code. It worked fantastically and really showed how these tools can be a boon.
I want to taste that same thrill again, but these days I'm lucky if I can get something out of it that will even compile, never mind the logical correctness. Maybe I'm just getting worse at using the tools.
I find Claude good at helping me find how to do things that I know are possible but I don’t have the right nomenclature for. This is an area where Google fails you, as you’re hoping someone else on the internet used similar terms as you when describing the problem. Once it spits out some sort of jargon I can latch onto, then I can Google and find docs to help. I prefer to use multiple sources vs just LLMs, partially because of hallucination, but also to keep amassing my own personal context. LLMs are excellent as librarians.
If you don't even understand your own PR, I'm not sure why you expect other people can.
I have used LLMs myself, but mostly for boilerplate and one-off stuff. I think it can be quite helpful. But as soon as you stop understanding the code it generates you will create subtle bugs everywhere that will cost you dearly in the long run.
I have the strong feeling that if LLMs really outsmart us to the degree that some AI gung-ho types believe, the old Kernighan quote will get a new meaning:
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it."
We'll be left with code nobody can debug because it was handed to us by our super smart AI that only hallucinates sometimes. We'll take the words of another AI that the code works. And then we'll hope for the best.
Coding is still a skill acquisition that takes years. We need to stamp out the behavior of not understanding what they take from copilot, but the behavior is not new.
It certainly has its uses - it's awesome at mocking and filling in the boilerplate unit tests.
Anything the difficult or complex, and it's really a coinflip if it's even an advantage, most of the time it's just distracting and giving irrelevant suggestions or bad textbook-style implementations intended to demonstrate a principle but with god-awful performance. Likely because there's simply not enough training data for these types of tasks.
With this in mind, I don't think it's strange that junior devs would be gushing over this and senior devs would be raising a skeptical eyebrow. Both may be correct, depending on what you work on.
But what I really appreciate is, I don't have to do the plug and chug stuff. Those patterns are well defined, I'm more than happy to let the LLM do that and concentrate on steering whether it's making a wise conceptual or architectural choice. It really seems to act like a higher abstraction layer. But I think how the engineer uses the tool matters too.
Personally I think that senior devs might fear a conflict within their identity. Hence they draw the 'You and the AI have no cue' card.
Where I do find it useful are
1) questions about frameworks/languages that I don't work in much and for which there is a lot of example content (i.e., Qt, CSS);
2) very specific questions I would have done a Google Search (usually StackOverflow) for ("what's the most efficient way to CPU and RAM usage on Windows using python") - the result is pointing me to a library or some example rather than directly generating code that I can copy/paste
3) boilerplate code that I already know how to write but saves me a little time and avoids typing errors. I have the CoPilot plugin for PyCharm so I'll write it as a comment in the file and then it'll complete the next few lines. Again best results is something that is very short and specific. With anything longer I almost always have to iterate so much with CoPilot that it's not worth it anymore.
4) a quick way to search documentation
Some people have said it's good at writing unit tests but I have not found that to be the case (at least not the right kind of unit tests).
If I had to quantify it, I'd probably give it a 5-10% increase in productivity. Much less than I get from using a full featured IDE like PyCharm over coding in Notepad, or a really good git client over typing the git commands in the CLI. In other words, it's a productivity tool like many other tools, but I would not say it's "revolutionary".
Books and manuals, they're pretty great for introductory materials. And for advanced stuff, you have to grok these first.
> 2) very specific questions I would have done a Google Search (usually StackOverflow) for ("what's the most efficient way to CPU and RAM usage on Windows using python")
I usually go backwards for such questions, searching for not what I want to do, but how it would look like if it exists. And my search-fu have not failed me that much in that regards, but that requires knowledge on how those things work, which again goes back to books and other such materials.
> 3) boilerplate code that I already know how to write but saves me a little time and avoids typing errors.
Snippets and templates in my editor. And example code in the documentation.
4) a quick way to search documentation
I usually have a few browser tabs open for whatever modules I'm using, plus whatever the IDE has, and PDFs and manual pages,...
For me, LLMs feel like building a rocketship to get groceries at the next village, and then hand-waving the risks of explosions and whether it would actually get you there.
So it's not like CoPilot is giving me information that I couldn't get fairly easily before. But it is giving it to me much __faster__ than I could access it before. I liken it to an IDE tool that allows you to look up API methods as you type. Or being able to ask an expert in that particular language/domain, except it's not as good as the expert because if the expert doesn't know something they're not going to make it up, they'll say "don't know".
So how much benefit you get from it is relative to how much you have to look up stuff that you don't know.
I've been using Cursor for around 10 days on a massive Ruby on Rails project (a stack I've been coding in for +13 years).
I didn't enjoy any productivity boost on top of what GitHub Copilot already gave me (which I'd estimate around the 25% mark).
However, for crafting a new project from scratch (empty folder) in, say, Node.js, it's uncanny; I can get an API serving requests from a OpenAPI schema (serving the OpenAPI schema via swagger) in ~5 minutes just by prompting.
Starting a project from scratch, for me at least, is rare, which probably means going back to Copilot and vanilla VSCode.
@workspace /new “scaffold a ${language} project”
I haven’t been able to get any mileage out of chat AI beyond treating it like a search engine, then verifying what it said…. Which isn’t a speedy workflow
- writing robust bash and using unix/macos tools
- how to do X in github actions
- which API endpoint do I use to do Y
- synthesizing knowledge on some topic that would require dozens of browser tabs
- enumerating things to consider when investigating things. Like "I'm seeing X, what could be the cause, and how I do check if it's that". For example I told it last week "git rebase is very slow, what can it be?" and it told me to use GIT_TRACE=1 which made me find a slow post-commit hook, and suggested how to skip this hook while rebasing.
Said hysteria was built on the same idea. After all, LLMs themselves are just compilers for a programming language that is incredibly similar to spoken language. But as the programming language is incredibly similar to the spoken language that nearly everyone already knows, the idea was that everyone would become the metaphorical elevator operator, "wiping out" programming as a job just as elevator operators were "wiped out" of a job when operating an elevator became accessible to all.
The key difference, and where the hysteria is likely to fall flat, is that when riding in an elevator there isn't much else to do but be the elevator operator. You may as well do it. Your situation would not be meaningfully improved if another person was there to press the button for you. When it comes to programming, though, there is more effort involved. Even when a new programming language makes programming accessible, there remains a significant time commitment to carry out the work. The business people are still best to leave that work to the peons so they can continue to focus on the important things.
But coding is just a fraction of my weekly workload, and AI has been less impactful for other aspects of project management.
So overall it’s 25%-50% increase in productivity.