Effects of Gen AI on High Skilled Work: Experiments with Software Developers

That's across the board, from the startups whose business plan is to be acquired at all costs, to the giant tech companies, whose business plan is to get monopoly power first, then figure out how to extract money later.

In my banking corp, in past 13 years I've seen massive rise of complexity, coupled with absolutely madly done bureaucracy increase. I still could do all stuff that is required but - I dont have access. I cant have access. Simple task became 10 steps negotiating with obscure Pune team that I need to chase 10x and escalate till they actually recognize there is some work for them. Processes became beyond ridiculous, you start something and it could take 2 days or 3 months, who knows. Every single app will break pretty quickly if not constantly maintained - be it some new network stuff, unchecked unix update, or any other of trillion things that can and will go wrong.

This means - paper pushers and folks at best average at their primary job (still IT or related) got very entretched in processes and won, and business gets served subpar IT, projects over time and thus budget, perpetuating the image of shitty tolerated evil IT.

I stopped caring, work to live is more than enough for me, that 'live' part is where my focus is and life achievements are.

Btw, they key skill you're leaving out, is to understand the business your company is in.

If you can couple even moderate developer ability with a good understanding of business objectives, you may stay relevant even while some of the pure developers are replaced by AI.

I think that what the MBAs miss is this phenomena of overconstraint. Once you have separate the generic role of "developer" into "developer, operations, and security", you've likely specified all sorts of details about how those roles need to be done. When you combine them back into DevSecOps, all the details remain, and you have one person doing 3x the work instead of one person doing the work 3x more efficiently. To effectively go backwards, you have to relax constraints, and let that one person exercise their judgment about how to do the job.

A corollary is that org size can never decrease, only increase. As more employees are hired, jobs become increasingly specialized. Getting rid of them means that that job function is simply not performed, because at that level of specialization, the other employees cannot simply adjust their job descriptions to take on the new responsibilities. You have to throw away the old org and start again with a new, small org, which is why the whole private equity / venture capital / startup ecosystem exists. This is also Gall's Law exists:

https://en.wikipedia.org/wiki/John_Gall_(author)#Gall's_law

What has changed is micromanaging of daily standup which reshapes work into neat packages for conversation but kills a non linear flow and limits exploration making things exactly as requested instead of what could be better.

Think of your "DevSecOps" people doing 3x the work they should. Do you know what else are they doing? Booking their own business travels. Reconciling and reporting their own expenses. Reporting their own hours, broken down by business categories. Managing their own vacation days. Managing their own meetings. Creating presentations on their own, with graphics they made on their own. Possibly even doing 80% of work on lining up purchases from third parties. And a bunch of other stuff like that.

None of these are part of their job descriptions - in fact, all of these are actively distracting and disproportionally compromise the workers' ability to do their actual jobs. All of these also used to have dedicated specialists, that could do it 10x as efficiently, for fraction of the price.

My hypothesis is this: those specialists like secretaries, internal graphics departments, financial staff, etc. they all were visible on the balance sheet. Eliminating those roles does not eliminate the need for their work to be done - just distributes it to everyone in pieces (in big part thanks to self-serve office software "improving" productivity). That slows everyone down across the board disproportionally, but the beancounters only see the money saved on salaries of the eliminated roles - the slowdown only manifests as a fuzzy, generic sense of loss of productivity, a mysterious costs disease that everyone seems to suffer from.

I say it's not mysterious; I say that there is no productivity gain, but rather productivity loss - but because it turns the costs from legible, overt, into diffuse and hard to count, it's easy to get fooled that money is being saved.

There's also no platform engineers but IaC has gotten that good that arguably they've become redundant. Architecture decisions get made on the fly by team members rather than by the Software Architect who only shows up now and again to point out something trivial. No Product Owner so again the team work out the requirements and write the tickets (ChatGPT can't help there).

It is so much easier to deploy now (and for the last 5-10 years) without managing an actual server and OS

It just gets easier, with new complexities added on top

In 2014 I was enamored that I didn’t need to be a DBA because platforms as a service were handling all of it in a NoSQL kind of way. And exposed intuitive API endpoints for me.

This hasn’t changed, at worst it was a gateway drug to being more hands on

I do fullstack development because it’s just one language, I do devops because it’s not a fulltime job and cloud formation scripts and further abstraction is easyish, I can manage the database and I haven’t gotten vendor locked

You don’t have to wait 72 hours for DNS and domain assignments to propagate anymore it’s like 5 minutes, SSL is free and takes 30 minutes tops to be added to your domain, CDNs are included. Over 10 years ago this was all so cumbersome

I think it entirely has to do with a generation of software people getting into the field (understandably) because it makes them a lot of money, rather than because they're passionate about software. These, by-and-large, are mediocre technical people and they tend to hire other mediocre technical people.

When I look around at many of the newer startups that are popping up they're increasing filled with really talented people. I chalk this up largely to the fact that people that really know what they're doing are increasingly necessary to get a company started in a more cash constrained environment and those people are making sure they hire really talented people.

Tech right now reminds me so much more of tech around 2004-2008, when almost everyone one that was interested in startups was in it because they loved hacking on technical problems.

My experience with Cursor is that it is excellent at doing things mediocre engineers can do, and awful at anything more advanced. It also requires the ability to very quickly understand someone else's code.

I could be wrong, but my suspicion is this will allow a lot of very technical engineers, that don't focus on things like front-end or web app development, to forgo needing to hire as many junior webdev people. Similar to how webmasters disappeared once we have frameworks and tools for quickly building the basic HTML/CSS required for a web page.

You can categorize get people who are average at several things devOps. But you will not get someone with a deep background and understanding in all the fields at the same time.

I come from a back end background. and I appalled at how little the devOps I have worked with know about even SQL.

Having teams with people who are experts at different things will give a lot better output. It will be more expensive.

Most devOps I have met, with a couple of exceptions, are front end devs who knows a couple of Javascript, knows Javascript and Typescript. When it comes to the back-end it is getting everything possible form npm and stringing it together.

So many companies no longer think about quality or design. “Just build this now and ship it, we can modify it later”, not thinking about the ramifications.

No thinking about design at all anymore, then having tech debt but not allocating any sprints to mitigate it.

I have received e-mails "hey thats DB don't touch that stuff you are not a DBA" or "hey developers shouldn't do QA" while the premise might be right, lots of things could be done much quicker.

I have seen throwing stuff over the wall "hey I am a developer, I don't care about some server stuff", "hey my DB is working fine it must be you application" or months of fixing an issue because no one would take the responsibility across the chain.

If fewer people need to undertake more roles, I think the simplest things you can get away with should be chosen, yet for whatever reason that's not what's happening.

Need a front end app? Go for the modern equivalent of jQuery/Bootstrap, e.g. something like Vue, Pinia and PrimeVue (you get components out of the box, you can use them, you don't have to build a whole design system, if needed can still do theming). Also simpler than similar setups with Vuex or Redux in React world.

Need a back end app? A simple API only project in your stack of choice, whether that's Java with Dropwizard (even simpler than Spring Boot), C# with ASP.NET (reasonably simple out of the box), PHP with Laravel, Ruby with Rails, Python with Flask/Django, Node with Express etc. And not necessarily microservices but monoliths that can still horizontally scale. A boring RESTful API that shuffles JSON over the wire, most of the time you won't need GraphQL or gRPC.

Need a database? PostgreSQL is pretty foolproof, MariaDB or even SQLite can also work in select situations. Maybe something like Redis/Valkey or MinIO/SeaweedFS, or RabbitMQ for specific use cases. The kinds of systems that can both scale, as well as start out as a single container running on a VPS somewhere.

Need a web server? Nginx exists, Caddy exists, as does Apache2.

Need to orchestrate containers? Docker Compose (or even Swarm) still exist, Nomad is pretty good for multi node deployments too, maybe some relatively lightweight Kubernetes clusters like K3s with Portainer/Rancher as long as you don't go wild.

CI/CD? Feed a Dockerfile to your pipeline, put the container in Nexus/Artifactory/Harbor/Hub, call a webhook to redeploy, let your monitoring (e.g. Uptime Kuma) make sure things remain available.

Architectures that can fit in one person's head. Environments where you can take every part of the system and run it locally in Docker/Podman containers on a single dev workstation. This won't work for huge projects, but very few actually have projects that reach the scale where this no longer works.

Yet, this is clearly not what's happening, that puzzles me. If we don't have 20 different job titles involved in a project, then the complexity covered under the "GlassFish app server configuration manager" position shouldn't be there in the first place (once had a project like that, there was supposed to be a person involved who'd configure the app server for the deployments, until people just went with embedded Tomcat inside of deployable containers, that complexity suddenly dissipated).

I found the tool to be extremely valuable when working in unfamiliar languages, or when doing rote tasks (where it was easy for me to identify if the generated code was up to snuff or not).

Where I think it falters for me is when I have a very clear idea of what I want to do, and its _similar_ to a bog standard implementation, but I’m doing something a bit more novel. This tends to happen in “reduce”s or other more nebulous procedures.

As I’m a platform engineer though, I’m in a lot of different spaces: Bash, Python, browser, vanilla JS, TS, Node, GitHub actions, Jenkins Java workflows, Docker, and probably a few more. It gives my brain a break while I’m context switching and lets me warm up a bit when I move from area to area.

For example, writing IaC especially for AWS, I have to look up tons of stuff. Asking AI gets me answers and examples extremely fast. If I'm learning the IaC for a new service I'll look over the AWS docs, but if I just need a quick answer/refresher, AI is much faster than going and looking it up.

A psychology professor I know says this holds in general. For any new tool, who will be able to get the most benefits out of it? Someone with a lot of skill already or someone with fewer skill? With less skill, there is even a chance that the tool has a negative effect.

But Claude Sonnet 3.5 w/ Cursor or Continue.dev is a dramatic improvement. When you have discrete control over the context (ie. being able to select 6-7 files to inject), and with the superior ability of Claude, it is an absolute game changer.

Easy 2-5x speedup depending on what you're doing. In an hour you can craft a production ready 100 loc solution, with a full complement of tests, to something that might otherwise take a half day.

I say this as someone with 26 yoe, having worked in principal/staff/lead roles since 2012. I wouldn't expect nearly the same boost coming at less than senior exp. though, as you have to be quite detailed at what you actually want, and often take the initial solution - which is usually working code - and refine it a half dozen times into something that you feel is ideal and well factored.

But recently using Claude Sonnet + Haiku through OpenRouter also with aider, and it is like a new dimension of programming.

Working on new projects in Rust and a separate SPA frontend, it just ... implements whatever you ask like magic. Gets it about 90-95% right at the first prompt. Since I am pretty new to Rust, there are a lot of idiomatic things to learn, and lots of std convenience functions I don't yet know about, but the AI does. Figuring out the best prompt and context for it to be effective is now the biggest task.

It will be crazy to see where things go over the next few years... do all junior programmers just disappear? Do all programmers become prompt engineers first?

The more you lean into functional patterns: design some monads, don’t do I/O except at the boundaries, use fluent programming, then it’s highly effective.

This is all in Java, for what it’s worth. Though, I’ll admit, I’m 3.5y into Java, and rely heavily on Java 8+ features. Also, heavy generic usage in my library code gives a lot of leash to the LLM to consistently make the right choice.

I don’t see these gains as much when using quicker/sloppier designs.

Would love to hear more from true FP users (Haskell, OCaml, F#, Scala).

So, like before AI then? I haven't seen AI deliver illogical nonsense that I couldn't even decipher like I have seen some outsourcing companies deliver.

Thank you, this says what I have been struggling to describe.

The day I lost part of my soul was when I asked a dev if I could give them feedback on a DB schema, they said yes, and then cut me off a few minutes in with, “yeah, I don’t really care [about X].” You don’t care? I’m telling you as the SME for this exactly what can be improved, how to do it, and why you should do so, but you don’t care. Cool.

Cloud was a mistake; it’s inculcated people with the idea that chasing efficiency and optimization doesn’t matter, because you can always scale up or out. I’m not even talking about doing micro-benchmarks (though you should…), I’m talking about dead-simple stuff like “maybe use this data structure instead of that one.”

This was done because we asked for a minor change to be done (talking maybe 5 lines of code) and tested. So now not only are we dealing with new debt, we are dealing with code that no one can explain why it was completely changed (and some of the changes were changes for the sake of change), and we are dealing with those of us that manage this code now looking at completely foreign code.

I keep seeing this with people that are using these tools and they are not higher level engineers. We finally got to the point of denying these PR's and saying to go back and do it again. Loosing any of the time that was theoretically gained from doing it in the first place.

Not saying these tools don't have a place. But people are using it without understanding what it is putting out and not understanding the long term effects it will have on a code base.

I found this too. But I also found the opposite, including here on HN; people who are interested in technology have almost an aversion against using AI. I personally love tech and I would and do write software for fun, but even that is objectively more fun for me with AI. It makes me far more productive (very much more than what the article states) and, more importantly, it removes the procrastination; whenever I am stuck or procrastinating getting to work or start, I start talking with Aider and before I know it, another task was done that I probably wouldn't have done that day without.

That way I now launch bi weekly open and closed source projects while before that would take months to years. And the cost of having this team of fast experienced devs sitting with me is max a few $ per day.

It reminds me of a talk Raymond Hettinger put on a while ago about rearranging the flowers in the garden. There is a tendency from new developers to rearrange for no good reason, AI makes it even easier now. This comes down to a culture problem to me, AI is simply the tool but the driver is the human (at least for now).

We use it internally and the technical debt is an enormous threat that IMO hasn't been properly gauged.

It's very very useful to carpet bomb code with APIs and patterns you're not familiar with, but it also leads to insane amounts of code duplication and unwieldy boilerplate if you're not careful, because:

1. One of the two big bias of the models is the fact that the training data is StackOverflow-type training data, which are examples and don't take context and constraints into account.

2. The other is the existing codebase, and it tends to copy/repeat things instead of suggesting you to refactor.

The first is mitigated by, well, doing your job and reviewing/editing what the LLM spat out.

The second can only be mitigated once diffs/commit history become part of the training data, and that's a much harder dataset to handle and tag, as some changes are good (refactorings) but other might be not (bugs that get corrected in subsequent commits) and no clear distinction as commit messages are effectively lies (nobody ever writes: bug introduced).

Not only that, merges/rebases/squashes alter/remove/add spurious meanings to the history, making everything blurrier.

Is this a bad thing? Maybe I'm misunderstanding it, but even when I'm working on my own projects, I'm usually trying to solve a problem, and the technology is a means to an end to solving that problem (delivering). I care that it works, and is maintainable, I don't care that much about the technology.

Bingo, this, so much this. Every dev i know who loves AI stuff was a dev that I had very little technical respect for pre AI. They got some stuff done but there was no craft or quality to it.

You get what you measure. Nobody measure software quality.

So, pretty similar to how it was before. Except that motivated junior developers will improve incredibly fast. But that's also kind of always been the case in software development these past two decades?

  > I wonder if the study includes the technical debt that more experienced developers had to tackle after the less experienced devs have contributed their AI-driven efforts. 

You also may find this post from the other day more illuminating[0], as I believe the actual result strongly hints at what you're guessing. The study is high schoolers doing math. While GPT only has an 8% error rate for the final answer, it gets the steps wrong half the time. And with coding (like math), the steps are the important bits.

But I think people evaluate very poorly when there's ill defined metrics but some metric exists. They over inflate it's value since it's concrete. Like completing a ticket doesn't mean you made progress. Introducing technical debt would mean taking a step back. A step forward in a very specific direction but away from the actual end goal. You're just outsourcing work to a future person and I think we like to pretend this doesn't exist because it's hard to measure.

[0] https://news.ycombinator.com/item?id=41453300

For them programming is a means to an end, and I think it is fine, in a way. But you cannot just ask an AI to write you tiktok clone and expect to get the finished product. Writing software is an iterative process, and LLMs currently used are not good enough for that, because they need not only to answer the questions, but at the very minimum to start asking questions: "why do want to do that ?" "do you prefer this or that", etc., so that they can actually extract all the specification details that the user happily didn't even know he needed before producing an appropriate output. (It's not too different from how some independent developers has to handle their clients, isn't it ?). Probably we will get there, but not too soon.

I also doubt that current tools can keep a project architecturally sound long-term, but that is just an hunch.

I admit though that I may be biased because I don't like much tools like copilot: when I write software, I have in my mind a model of the software that I am writing/I want to write, the AI has another model "in mind" and I need to spend mental energy understanding what it is "thinking". Even if 99/100 it is what I wanted, the remaining 1% is enough to hold me back from trusting it. Maybe I am using it the wrong way, who knows.

The AI tool that work for me would be a "voice controller AI powered pair programmer": I write my code, then from time to time I ask him questions on how to do something, and I can get either an contextual answer depending on the code I am working on, or generate the actual code if I wish so". Are there already plugins working that way for vscode/idea/etc ?

If I don't deliver my startup burns in a year. In my previous role if I didn't deliver the people who were my reports did not get their bonuses. The incentives are very clear, and have always been clear - deliver.

The few attempts I've made at using genAI to make large-scale changes to code have been failures, and left me in the dark about the changes that were made in ways that were not helpful. I needed suggestions to be in much smaller chunks. paragraph sized. Right now I limit myself to using the genAI line completion suggestions in Pycharm. It very often guesses my intentions and so actually is helpful, particularly when laboriously typing out lots of long literals, e.g. keys in a dictionary.

You are not a fucking priest in the temple of engineering, go to fucking CS dep at the local uni and be the one and preach it there. You are worker of the company with customers, which pays you a salary from customers money.

Accenture is the kind of company that cooperates and co-markets with large orgs like Microsoft. With ~300 devs in the pool they hardly move the population at all, and they cannot be assumed to be objective since they are building a marketing/consulting division around AI workflows.

The third anonymous company didn’t actually have a randomized controlled trial, so it is difficult to say how one should combine their results with the RCTs. Additionally, I am sure that more than one large technology company went through similar trials and were interested in knowing the efficacy of them. That is to say, we can assume other data exist than just those included in the results.

Why did they select these companies, from a larger sample set? Probably because Microsoft and Accenture are incentivized by adoption, and this third company was picked through p-hacking.

In particular, this statement in the abstract is a very bad sign:

> Though each separate experiment is noisy, combined across all three experiments

It is essentially an admission that individually, the companies don’t have statistically significant results, but when we combine these three (and probably only these three) populations we get significant results. This is not science.

There is skill involved in using generative code models and it’s the same skill you need for delegating work to others and integrating solutions from multiple authors into a cohesive system.

At least Copilot you’re still more or less driving, whereas ChatGPT you’re more the passenger and not growing intuition.

I share your hunch, though I would go so far as to call it an informed, strong opinion. I think we're going to pay the price in this industry in a few years, where the pipeline of "clueful junior software developers" is gonna dry way up, replaced by a firehose of "AI-reliant junior software developers", and the distance between those two categories is a GULF. (And of course, it has a knock-on effect on the number of clueful intermediate software developers, and clueful senior software developers, etc...)

But for folks who are going to be successful with or without it, it's a godsend in terms of being able to essentially ask stack overflow questions and get immediate non judgemental answers.

Maybe not correct all the time, but that was true with stack overflow as well. So as always, it comes back to the individual.

How many contemporary developers have no idea how to write machine code, when 50 years ago it was basically mandatory if you wanted to be able to write anything?

Are LLM's just going to become another abstraction crutch turned abstraction solid pillar?

Well, at least that's how I use them. And to throw a counter to your hypothesis, I find that sometimes the LLM will use functions or library components that I didn't know of, which actually saves me a lot of time when learning a new language or toolkit. So for me, it actually accelerates learning rather than retarding it.

When used scientifically, coding copilots boost productivity AND skills.

E.g last night I setup my first Linux raid. A task that isn't too hard, but following a tutorial or just "reading the docs" isn't particularly helpful given it takes a few different tools (mount, umount, fstab, blkid, mdadm, fdisk, lsblk, mkfs) and along the way things might not follow the exact steps from a guide. I asked dozens of questions about each tool and step, where previously I would have just "copy paste and prayed".

Two nights ago I was similarly able to fully recover all my data from a failed ssd also using chatgpt to guide my learning along the way. It was really cool to tackle a completely new skill having a "guide" even if it's wrong 20% of the time, that's way better than the average on the open Internet.

For someone who loves learning, it feels like thousand league boots compared to just endlessly sifting through internet crap. Of course everything it says is suspect, just like everything else on the Internet, but boy it cuts out a lot of the hassle.

I have seen mostly senior programmers argue why ai tools don't work. Juniors just use them without prejudice.

I've been using Cursor for around 10 days on a massive Ruby on Rails project (a stack I've been coding in for +13 years).

I didn't enjoy any productivity boost on top of what GitHub Copilot already gave me (which I'd estimate around the 25% mark).

However, for crafting a new project from scratch (empty folder) in, say, Node.js, it's uncanny; I can get an API serving requests from a OpenAPI schema (serving the OpenAPI schema via swagger) in ~5 minutes just by prompting.

Starting a project from scratch, for me at least, is rare, which probably means going back to Copilot and vanilla VSCode.

I'm not great at remembering specific quirks/pitfalls about secondary languages like e.g. what the specific quoting incantations are to write conditionals in Bash, so I rarely wrote bash scripts for automation in the past. Basically only if that was a common enough task to be worth the effort. Same for processing JSON with jq, or parsing with AWK.

Now with LLMs, I'm creating a lot more bash scripts, and it has gotten so easy that I'll do it for process-documentation more often. E.g. what previously was a more static step-by-step README with instructions is now accompanied with an interactive bash script that takes user input.

Where I do find it useful are

1) questions about frameworks/languages that I don't work in much and for which there is a lot of example content (i.e., Qt, CSS);

2) very specific questions I would have done a Google Search (usually StackOverflow) for ("what's the most efficient way to CPU and RAM usage on Windows using python") - the result is pointing me to a library or some example rather than directly generating code that I can copy/paste

3) boilerplate code that I already know how to write but saves me a little time and avoids typing errors. I have the CoPilot plugin for PyCharm so I'll write it as a comment in the file and then it'll complete the next few lines. Again best results is something that is very short and specific. With anything longer I almost always have to iterate so much with CoPilot that it's not worth it anymore.

4) a quick way to search documentation

Some people have said it's good at writing unit tests but I have not found that to be the case (at least not the right kind of unit tests).

If I had to quantify it, I'd probably give it a 5-10% increase in productivity. Much less than I get from using a full featured IDE like PyCharm over coding in Notepad, or a really good git client over typing the git commands in the CLI. In other words, it's a productivity tool like many other tools, but I would not say it's "revolutionary".

But coding is just a fraction of my weekly workload, and AI has been less impactful for other aspects of project management.

So overall it’s 25%-50% increase in productivity.

- writing robust bash and using unix/macos tools

- how to do X in github actions

- which API endpoint do I use to do Y

- synthesizing knowledge on some topic that would require dozens of browser tabs

- enumerating things to consider when investigating things. Like "I'm seeing X, what could be the cause, and how I do check if it's that". For example I told it last week "git rebase is very slow, what can it be?" and it told me to use GIT_TRACE=1 which made me find a slow post-commit hook, and suggested how to skip this hook while rebasing.

Copilot - “okay I can do that for you! Here are your new commits!”

Senior Dev - “why? Your change is atomic. I’ll tell management to f-off, in a kind way, if they bring up those silly change-per-month metrics again”

For public facing projects - your documentation just became part of the LLM's training data, so its now extra important your documentation is thorough and accurate because you will have a ton of developers getting answers from that system.

For private projects, your documentation can now be fed into a finetuning dataset or a RAG system, achieving the same effect.

1) You need to be the boss with the AI being your assistant. You are now a project manager coming up with strict requirements of what you'd like done. Your developer (AI) needs context, constraints and needs to be told exactly what you'd like created without necessarily diving into the technical details.

2) Planning - you need to have a high level plan of roughly how you'd like to structure your code. Think of it like you're drawing the outline and AI is filling in the gaps.

3) Separation of concerns - use software principles to drive your code design. Break problems down into separate components, AI is good at filling in components that are well defined.

Once you change your thinking to a higher level, then you can maintain flow state. Of course the AI isn't perfect and will make mistakes - you do need to question it as you go. The more creative you become with a solution the harder time the AI will have and sometimes you'll have to break out and fix things up yourself.

For me it's a huge productivity boost.

Let's not forget that developers some times do this, too...

Must have seen AI write the implementation as well?

If you're still cognizant of what you're writing on the implementation side, it's pretty hard to see a test go from failing to passing if the test is buggy. It requires you to independently introduce the same bug the LLM did, which, while not completely impossible, is unlikely.

Of course, humans are prone to not understanding the requirements, and introducing what isn't really a bug in the strictest sense but rather a misfeature.

"However, the table also shows that for all outcomes (with the exception of the Build Success Rate), the standard deviation exceeds the pre-treatment mean, and sometimes by a lot. This high variability will limit our power in our experimental regressions below."

What I find even stranger is that the values in the "control" and "treatment" columns are so similar. That would be highly unlikely given the extreme variability, no?

What's next? Royalties from companies that used the answer in their solutions? Do you also want copyright of this comment?

The experiment in question was to split 95 devs into two groups and see how long it took each group to setup a web server in Javascript. Control took a little under 3 hours on average, the copilot group took 1 hour and 11 minutes on average.

https://github.blog/news-insights/research/research-quantify...

And it is thanks to this weak experiment that Github proudly boasts that Copilot makes devs 55% faster.

By contrast the conclusion that Copilot makes devs ~25% more productive seems reasonable, especially when you read the actual paper and find out that among senior devs the productivity gains are more marginal.

Similar to gaming the stock price for a couple quarters as C-level, but now with more incentive for this sort of behavior at IC level

I'm actually fairly senior (turning 50 next month) and I notice an effect that AI is having on my own productivity: I now take on mini projects that I used to delegate or avoid doing because they would take too much time or be too tedious. That's not the case anymore. The number of things I can use chat gpt for is gradually expanding. I notice that I'm skilling up a lot more rapidly as well.

This is great because if you want to stay relevant, you need to adapt to modern tools and technology. That's nothing new of course. Changes are a constant in our industry. And there always are a lot of people that learn some tricks when they are young and then never learn anything new again. If you are lucky some of that stuff stays relevant for a few decades. But mostly a lot of stuff gets unceremoniously dumped by younger generations.

The ability to use LLMs is becoming an important skill in itself and one that is now part of what I look for in candidates (including older ones). I don't have a lot of patience for people refusing to use tools that are available to them. Tell me how you use tools to your advantage; not how you have a tool aversion or are unwilling to learn new things.

Is this because they are not using coding assistants? Are they resistant to using them? I have to say that the coding assistant is helpful; it is an ever-present rubber duck that can talk back with useful information.

Unless one is an absolutely terrible developer working on incredibly simple problems, this is simply impossible.

Potentially sharing company IP with a 3rd party?