Measuring the productivity impact of generative AI
nber.org
nber.org
Have there been any good studies on programmer productivity? 14% would be a just a blip in the constant flow of productivity-enhancing language, tools, architectures etc. that have been steady accelerating the pace and scale of development for decades ...
When I moved from Java to Rails, for example ... definitely at least 14% more productive, I'm guessing much much higher. Even just from load times.
Or going from printed documentation all the way to Stack Overflow - how many multiples was that?
68000 to M2 ... page reload to React's fast refresh ... etc. etc.
On the other hand, "zero or small negative effects on the most experienced/most able workers".
In other words, if you're skilled, there's nothing AI can do for you…
Yet...
Think about that…
I don't think you grasp how this will play out over multi-game
What matters is which of these strategies will result in actual success. This is honestly far too hard to tell at this point.
I'll say here, I've spoken to and spent time with a few well known billionaires and I'd say deep down, they are exterminists no matter how nice a face they try to put on it. Over time they just come to believe most people don't matter at all. Its really dark, and the sooner we come to terms with that the better.
While I agree with your assessment on wage pressure, the folks this is going to hurt the most are the new graduates that don't have the knowledge or experience. Their competition just got a whole lot stiffer.
It benefits two groups the most, someone with literally zero experience and experts.
In practice, my coworkers call me to find and then explain solutions to them.
I don’t see this changing with AIs. Having a “super Google” is great… for people already comfortable with such tools and capable of using them.
My coworkers will now ask me to ask the AI… on their… behalf.
OMG! I’m going to turn into Lieutenant Tawny Madison: https://chat.openai.com/share/0efbca4d-9e33-4af0-9ed6-2f8821...
Just think of when Photoshop 4.0 added layers and now composing was for everyone, not just for those venturing into channel operations… (To be fair, here, things got much easier even for those who managed previously without this.)
To be concrete about it, I recently ported a project from SwiftUI on iOS (an ecosystem I am very comfortable with) to an Android app written in Kotlin (an ecosystem I mostly dread). I don't find ChatGPT very helpful with my day-to-day Swift stuff, but it was incredibly helpful with the Android work, from language syntax to idioms to outright translation of code.
I tried using the Bing chat interface, and it repeatedly pushed me in the direction of using Microsoft tools to accomplish tasks for a cross-compile mid tier solution. I had to explicitly tell it to exclude them.
In areas where I'm an expert, I find myself correcting ChatGPT constantly. But for a language I want to learn, it's been really helpful to get me started on a simple algorithm or debug an error message.
If the task you are working on is phone support. If you are a dev, you are perpetually learning new things, it's not possible to memorise the whole field so AI would have more opportunity to help.
> "Customer support agents using an AI tool to guide their conversations saw a nearly 14 percent increase in [overall] productivity, with 35 percent improvements for the lowest skilled and least experienced workers, and zero or small negative effects on the most experienced/most able workers... [out of] 5,000 agents working for a Fortune 500 software company."
AI quality is somewhere in the middle between highly skilled and neophyte (at present anyway).
This coming century is supposed to be about 'Rest of World' - first China, then India and the rest.
But it's going to risk the ability of the nearly-unskilled labour to help.
They can learn new things and cover over the things they do now know. Hopefully they can join the Developed World quickly.
Every time someone says X is going to suffer under ChatGPT, I remind that person that X will also improve their productivity by using ChatGPT.
ChatGPT is the most amazing personal tutor I can find. I'm adding 1 IT cert per month, largely due to ChatGPT's help as my personal tutor.
Do you know what happened to manufacturing when automation low-cost outsourcing happened?
It was wiped out.
It was devastating for certain sectors of the economy, even as 'net productivity rose' - the surpluses were acquired by some, not others.
A lot of 'ghettos' in the US are a direct result of mass factory closures.
Now - imagine that happening over the developing world, or rather, factories that were supposed to open, never did.
The developing world are 'services export' economy, with things like call centres etc. - and AI will more likely than not just evaporate those roles.
That those people will have 'access to ChatGPT' is besides the point when most of them don't even have computers (just mobile phones), or any way to apply that knowledge.
It's a bit like saying: 'The developing world has access to Wikipedia and all of Harvard courses online! They should all have great jobs!'
Unfortunately that's not how it works.
AI is going to help white collar workers, not pink or blue collar work which is low-skilled.
Elaborating further on the idea of productivity, as I said above, this doesn’t make employees more productive. It is doing some of the producing. It is not like an improved tool, where someone has to operate it to reap the benefits. It “upgrades productivity” more like how a kiosk at a McDonald’s upgrades productivity - not of the worker, but of the business. And much like the kiosks, we would be naive to believe they won’t replace the humans in due time.
This also has larger consequences for knowledge workers. Previously, to attain a higher level of productivity out of a worker, companies would have to invest in them so they could develop the necessary skills. Gone (or at least lessened) is that need. So workers will have less skill, less job freedom, a smaller share of profit, less social mobility. This is a nightmare for the working class.
Absolutely no one is spending tens of millions of dollars developing these tools in order to let random plebs capture much of the value produced by them.
AI will further consolidate mega corp power to a terrifying degree.
Consider the relationship between atmospheric nuclear testing and low-background steel [1] but for AI and older workers whose knowledge predates the introduction of LLMs in the workforce.
My guess though is that high skilled workers remain far more productive than AI-augmented low skilled workers.
Yes, but mostly from the companies developing these products:
* The CoPilot productivity study by Peng et al. that is summarized in a Blog post [1] with some additional details in the Arxiv preprint [2]. They find "developers who used GitHub Copilot completed the task significantly faster–55% faster than the developers who didn’t use GitHub Copilot". (Grep for "but speed is important, too" in [1].)
* Amazon's Q1 earnings report [3] includes a paragraph stating that "participants [using] CodeWhisperer completed tasks 57% faster (on average) and were 27% more likely to complete them successfully than those who didn’t use CodeWhisperer.
* I seem to remember seeing something on Replit's blog a while back with a similar number, but can't find it anymore, so maybe I'm mistaken.
These speed numbers are on specific and well-specified programming tasks, which is of course only one part of a developer's job, so there's a lot of opportunity for impactful research in this space. I suspect that the Empirical Software Engineering community will be awash in third-party empirical studies asking all sorts of more specific productivity questions by this time next year.
[1] https://github.blog/2022-09-07-research-quantifying-github-c...
[2] https://arxiv.org/pdf/2302.06590.pdf
[3] https://s2.q4cdn.com/299287126/files/doc_financials/2023/q1/...
> The tool was rolled out to the agents gradually, mostly between November 2020 and February 2021.
So the 14% gain came from vastly less capable systems than what we now have today.
When copilot was released, I’d say I got a 15% increase. When ChatGPT was released it was like 50% at least and I can’t imagine going back. I remember how slow it was now.
My advice would be to force yourself to leverage it more or something. I hate googling now. I’ll find a page, copy the entire thing into gpt4 and the file i’m using with the error message and i have to do nothing.
How are y’all using it?
> I’ll find a page, copy the entire thing into gpt4 and the file i’m using with the error message and i have to do nothing.
can you explain more about this I don't quite understand this passage.
Voice support will begin replacement or augmentation with AI once people are comfortable talking to a competent AI with good voice synthesis & response latency drops to milliseconds
Not good for senior agent wages short term, only good for senior agents long term when only hiring people for escalations. Likely a large reduction of the workforce in these roles overall.
I could do insane things with a hypothetical version of GPT 4 that’s private, effectively free, and a few orders of magnitude faster.
“Find a security vulnerability in this” thrown at a million lines of code could be fun.
It's linear per dollar but the price is also increasing, and that's specifically for customer hardware. Top end is still growing.
As for new algorithms, surely this would be more of a logarithmic increase for any specific problem as it gets harder and harder to improve on existing knowledge. Has, say, sorting become exponentially faster since Tony Hoare invented quicksort in 1959? My guess is no.
They're like a CPU: highly programmable, flexible, multi-purpose. This means that when any single "part" of the GPT inference code is running, it's probably using only 10-40% of the die area while the rest is not used and idle.
Imagine a future where a nearly-optimal architecture for LLMs has been figured out and stops changing every few months. At that point, it would be possible to make a dedicated chip that does "nothing else but that". It would be 100% utilised and faster because it would have more units dedicated to the specific task at hand. The data paths could also be optimised to minimise wasteful trips to-and-from external memory, etc...
The ideal "inference chip" would be something akin to an FPGA. Huge numbers of simple cells that can do just the dozen-or-so instructions required, with local memory for the weights and biases. Make these on a cheaper process, run them at 500 MHz for power efficiency, and tile them out by their hundreds in a hybrid memory+compute architecture.
It would be relatively easy to get a factor of 10x, and I suspect 100x is possible.
You are describing a privately hosted LLaMA, Falcon or Starcoder instance (albeit with lower quality than GPT4). You pay for the speed you want, though TBH they are already very fast on modest GPUs.
I’d be more curious to see the benefit vs good non AI UX
I think this line explains the study results quite well. In a closed domain like customer support for a specific product you are bound to uncover patterns that inexperienced employees can copy. It is also kind of limited by itself as as you progress you will inevitably stop relying on it.
I think this sort of tool can have a great impact for onboarding in these domains but other than that, this and personal experience seem to show that open ended knowledge work seems to only benefit marginally in the moment.
The most interesting thing to me is how little of a productivity boost you are seeing among higher skilled agents. Top quintile is getting an expected 2% boost with no change in the confidence interval.
This jives with a lot of the heterogeneous commentary you see on HN among engineers. GPT-3/4 are a godsend when approaching new systems; the productivity gain is quite small if you are already skilled.
It's interesting to me why these tools are actually useful. I've been at this for a while. I used to use the web in general, and various community forums to help advance my knowledge. Comp.lang.* and mailing lists were so helpful. And they were very approachable (low cost and multiple client tools). In it's early days, before the organization nazis showed up with more focus on rules than helpfulness, StackOverflow was that way. But at this point the internet feels like one of the tourist bazaars when I visited Jamaica (loved Jamaica). It used to feel more like a public library. Various attempts to monetize/fund various things, an explosion in options, have all rendered the internet far less usable. It used to be so much easier to find answers to things I know are possible, but don't remember the exact syntax for (e.g. "how do I test for directory existence in a bash script?"). To some degree, I view the rise of these GAs as a condemnation of what the internet has become from its original aspirations.
Maybe the whole "people will need UBI because robots will replace all the marginal workers" argument will fail because marginally productive workers will get so much better at their jobs.
In reality, automation and mechanization of manufacturing got rid of the skilled craftsman. Using the assembly line, Ford replaced artisans with people who were much less skilled.
What is important to consider though is that in this case here, the AI tool in question merely provided pre-written responses for agents to accept/reject/adapt, and how the effects are distributed may well shift as tools become more sophisticated and will just automatically handle most low-stakes/simple cases.
If 5000 call center agents saw an average increase in productivity by 14%, that means the company now needs 700 people less to handle all incoming calls.
I wouldn't be surprised if that 14% goes up to over 50% soon. In the past, I often asked on IRC when I had a question only someone with deep knowledge about a technology could answer. Nowadays I get a better answer from ChatGPT in over 50% of these cases.
>In Panel E, we see no discernible differences in surveyed customer satisfaction among pre- and post-AI groups
A call center's goal is to find the right spot along the cost/satisfaction curve. You can pay more money to answer calls quickly and increase satisfaction, or a smaller amount of money to answer calls slowly with lower satisfaction. More satisfied customers are more likely to spend more money with you, so as an operator you're trying to find the level of support that optimizes your bottom line.
If AI reduces the cost per call resolution, you're shifting that cost/satisfaction curve. Unless the wait-time to satisfaction relationship is perfectly linear, the new optimum will result in not firing 100% of the workers who could be "replaced" by productivity gains. You'll instead keep some of them around, with a lower total cost, and higher total customer satisfaction.
You'll still reduce staffing, but probably less than the raw productivity gain number.
14% is good during focus time but it's interesting that this results in "only" 1.3% overall increase. One potential limiting factor that comes to mind is the energy of the support rep. Doing 14% more during the same time might result in more fatigue, more recovery time, smaller increase in overall productivity. I also consider if there's some other process bottleneck downstream but that's a thought that stuck out to me.
edit: Reading it again, I see "successfully" resolved. Maybe it's as boring as incomplete data.
That said, tools like GH Copilot certainly gave me at least 10% boost. With the code often being repetitive (but not similar enough to e.g. be extracted to a separate function) the auto-completions definitely save a lot of time.
While I'm sceptical as to how far LLMs will go, AI as a whole (probably with different architecture in the future) will definitely go far beyond any productivity improvements currently possible.
While this does mean people tending to give up and leave support-queues is profitable, in the end it has degenerative impact on the corporate sales.
A monopolistic firm might be able to irritate customers this way, but it will not change cultural attitudes.
Have a great day friend, and remember misery is only a commodity in doomed institutions. =)
Unfortunately discourse around this particular topic tends to be less nuanced than it should be.
It's all about leverage.
Gains in efficiency only occasionally result in less jobs, generally they result in more production.
This is about the worst analogy that you could come up with.
https://www.fastcompany.com/3034289/your-car-centric-city-is...
Maybe a better example: Stick framing didn't eliminate pole barn construction.
Or prefab didn't eliminate build on site.
Being that it's an application that's really only a few months/years old it would be rather insane and worrying if we did replace people that fast. A better way to look at it would be farming in the US. In the 1850s we went from a very large percentage of people working in and around feeding the population. Today that number is rather small. This change lead to a massive amount of people moving from rural areas and a further increase in industrialization and factory workers.
The question we have today is, ok, we won't eliminate all humans in jobs, but what percent will we eliminate? How fast will it happen? What meaningful labor that actually pays enough for people to live will crop up in its place? If we look back at the history of the 1800s industrial revolution it was a very unstable time. There were labor wars in the US over unionization. Do we want to blindly stumble into another age of instability because of rapid changes in the labor market?
Never forget, it's not a companies job to have as many employees as possible, it's just currently that correlates with how much money they make. It is completely possible for AI to tip this balance where companies make huge profits, but those profits are tightly held and not distributed to a large labor pool. Income inequality is a huge problem in a world with expensive assets.
Jokes aside, when it comes to programming this stuff will actually have even bigger boosts for people who really understand how all the pieces fit together. It'll make gluing stuff up much less tedious. 14% seems way understated.
In cutting edge research there's probably a few hundred people in the world and the rest of us are in for the ride.
IMHO generative AI is an automated way of generating bullshit, which is fine as long as the data used for training is not bs itself you can probably derive some value in a limited set of contexts. Once you have the BS make it into the wild and it is fed back into the generator we will enter a BS downspiral.
If you're old enough to remember the world before search engines and how awesome search engines were in the beginning you've already seen this in action. Try using a search engine today. SEO spam made them almost unusable with really bad results.
This is the future of generative "AI". Show it work really really well for a small set of cases, get the imagination going and you're going heads first into a situation involving and emperor and lack of clothes.
The AI hype is also not new. !e've had several cycles of "thinking machines" taking over. I don't see how this time it's different.
It's always something that gets hyped up and gets the hustlers going: vr, ml, selfdriving, crypto, llms. Look at the tech and how it works, use it, ignore the hype.
This is already shown to not be an issue in research from the past year. It is possible to synthesize high quality data that still provides benefit when subsequently trained on.
It makes sense when you remember that this is what organic intelligence was doing all along.
Almost statistical noise