Anthropic Economic Index report: economic primitives
anthropic.com
anthropic.com
I expected to see measures of the economic productivity generated as a result of artificial intelligence use.
Instead, what I'm seeing is measures of artificial intelligence use.
I don't really see how this is measuring the most important economic primitives. Nothing related to productivity at all actually. Everything about how and where and who... This is just demographics and usage statistics...
>Instead, what I'm seeing is measures of artificial intelligence use.
Fun fact: this is also how most large companies are measuring their productivity increases from AI usage ;), alongside asking employees to tell them how much faster AI is making them while simultaneously telling them they're expected to go faster with AI.
In my experience, "good management" meant striving to isolate measurements as much as possible to output/productivity.
https://ourworldindata.org/grapher/labor-productivity-per-ho...
When you try and break it down to various products and cost centers is where it comes unstuck. It’s hard to impossible to measure the productivity of various teams contributing to one product, let alone a range of different products.
Moving fast and breaking things, agile.
On the other hand. When you know what you want to build but it’s a very large endeavor that takes careful planning and coordination across departments, traditional waterfall method still works best.
You can break that down into an agile-fall process with SAFe and Scrum of Scrums and all that PM mumbo jumbo if you need to. Or just kanban it.
In the end it’s just a mode of working.
In general, delaying infrastructure decisions as much as possible in process usually yields better infrastructure because the farther you are the more knowledge you have about the problem.
...that being said I do dislike how agile gets used as excuse for not doing any planning where you really should and have enough information to at least pick direction.
This is obviously satire but there's a clear ask, some features, from there you know what you need to have to even achieve those features, what project management process would you employ? Agile? Waterfall? Agile-fall? Kanban? Call me in 6 months?
Replacing existing system (and especially one you didn't write) is pretty much always the hardest case.
Any organization that properly adopted computers found out quickly how much they could improve productivity. The limiting factor was always understanding.
The trouble with AI tools is they don’t have this trajectory. You can be very versed on using them well, know all the best practices and where they apply and you get at best uneven gains. This is not the introduction of desktops 2.0
They define primitives as "simple, foundational measures of how Claude is used". They're not signing up to measure productivity, which would combine usage with displacement, work factoring, and a whole host of things outside their data pool.
What's the point? They're offering details on usage patterns relative to demographics that can help people assessing Anthropic's business and the utility of LLM-based AI. Notably, tasks and usage are concentrated in some industries (notably software) and localities (mainly correlated with GDP and the Gini index). This enables readers to project how much usage growth can be expected.
As far as I know, no one publicly offers this level of data on their emerging businesses - not google, ebay, apple, microsoft, amazon, nvidia or any of the many companies that have reshaped our technical and economic landscape in the last 30 years.
Normally we measure value with price and overall market (productivity gains is but one way that clients can recoup their price paid). But during this the build-out of AI, investors (of all stripes) are subsidizing costs to get share, so until we have stable competitive markets for AI services, value is an open question.
But it's clear some businesses feel that AI could be a strategic benefit, and they don't want to be left behind. So there is a stampede, as reflected in the neck-and-neck utilization of chat vs API.
At the very least for programming the decline of improvement compared to difficulty of task and skill of programmer quickly got into "AI technically helps but time spent fudging with prompts is more than time saved".
However if your task is to write some SEO slop about your products for your company's webpage, or to make a bunch of business emails to customers, I can absolutely believe AI can double productivity there, there is nothing of value produced that requires skill, it's just a few bits of info filled with fluff around them
I know what you mean.
Imagine my disappointment when I was expecting their unique approach and brainpower to have arrived at a straightforward index of overall world macroeconomic conditions rather than an internal corporate outlook for AI alone.
* value seems highly concentrated in a sliver of tasks - the top ten accounting for 32%, suggesting a fat long-tail where it may be less useful/relevant.
* productivity drops to a more modest 1-1.2% productivity gain once you account for humans correcting AI failure. 1% is still plenty good, especially given the historical malaise here of only like 2% growth but it's not like industrial revolution good.
* reliability wall - 70% success rate is still problematic and we're getting down to 50% with just 2+ hours of task duration or about "15 years" of schooling in terms of complexity for API. For web-based multi-turn it's a bit better but I'd imagine that would at least partly due to task-selection bias.
You can't compare the speed of AI improvements to the speed of technical improvements during the industrial revolution. ChatGPT is 3 years old.
The main difference is that people had no idea of the disruption it would cause and of course there wasn't there a huge investment industry around it.
The only question is about ROI of the investors will be positive (which depends on the timeline), not whether it is disruptive (or it will be after for example 30 years from now), and I see people confusing the two here quite often.
If the output of the model depends on the intelligence of the person picking outputs out of its training corpus, is the model intelligent?
This is kind of what I don't quite understand when people talk about the models being intelligent. There's a huge blindspot, which is that the prompt entirely determines the output.
In my experience with many PhDs they are just as prone to getting off track or using their pet techniques as LLMs! And many find it very hard to translate their work into everyday language too...
And?
These things are supposed to have intelligence on tap. I'll imagine this in a very simple way. Let's say "intellignce" is like a fluid. It's a finite thing. Intelligence is very valuable, it's the substrate for real-world problem solving that makes these things ostensibly worth trillions of dollars. Intelligence comes from interaction with the world; someone's education and experience. You spend some effort and energy feeding someone, clothing them, sending them to college. And then you get something out, which is intelligence that can create value for society.
When you are having a conversation with the AI, is the intelligence flowing out of the AI? Or is it flowing out of the human operator?
The answer to this question is extremely important. If the AI can be intelligent "on its own" without a human operator, then it will be very valuable -- feed electricity into a datacenter and out comes business value. But if a model is only intelligent as someone using it, well, the utility seems to be very harshly capped. At best it saves a bit of time, but it will never do anything novel, it will never create value on its own, independently, it will never scale beyond a 1:1 "human picking outputs".
If you must encode intelligence into the prompt to get intelligence out of the model, well, this doesn't quite look like AGI does it?
I see it with coworkers all the time. They'll ask ChatGPT to do an analysis and it'll output test results for a T-test. They don't know how to interpret it at all, and so it's ultimately meaningless to them. They're just using "stat sig" as a way to make a non-technical VP happy. In situations like this, I don't think a highly intelligent source, model or human, can make the recipient be more intelligent than they actually are.
You spend energy distilling the intelligence of the entire internet into a set of weights, but you still had to expend the energy to have humans create the internet first. And on top of this, in order to pick out what you want from the corpus, you have to put some energy in: first, the energy of inference, but second and far more importantly, the energy of prompting. The model is valuable because the dataset is valuable; the model output is valuable because the prompt is valuable.
So wait then, where does this exponential increase in value come from again?
I don't understand the analogy. A lever doesn't give you an increase in power (which would be a free lunch); it gives you an increase in force, in exchange for a decrease in movement. What equivalent to this tradeoff are you pointing to?
If you ask a sophisticated question (lots of clauses, college reading level or above) it will respond in kind.
You are basically moving where the generation happens in the latent space. By asking in a sophisticated way you are moving the latent space away from say children's books and towards say PhD dissertations.
Come on, this is human behavior 101, y’all.
you could argue that our input (senses) entirely define the output (thoughts, muscle movements, etc)
I just skimmed but is there any manual verification / human statistical analysis done on this or we just taking Claude’s word for it?
> a sustained increase of 1.0 percentage point per year for the next ten years would return US productivity growth to rates that prevailed in the late 1990s and early 2000s
What can it be compared to? Is it on the same level of productivity growth as computers? The internet? Sliced bread?
It also made me notice how much attention I’ve been giving these tech companies, almost as a substitute for the social media I try to avoid. I remember being genuinely excited for new posts on distill.pub the way I’d get excited for a new 3Blue1Brown or Veritasium video. These days, though, most of what I see feels like fingers-tired-from-scrolling marketing copy, and I can’t bring myself to care.
We get it guys the very scary future is here any minute now and you’re the only ones taking it super seriously and responsibly and benevolently. That’s great. Now please just build the damn thing
Note the papers cited are nearly all ones about AI use, and align more closely with management case studies vs. economics.
oh I know this one!
it's created mountains of systemic risk for absolutely no payoff whatsoever!
I would never make the argument that there are no risks. But there's also no way you can make the argument there are no payoffs!
Now you can’t be paid to do that task and you’ve made yourself unemployed or paid lower in the future.
Meanwhile, mountains of additional energy is being comsumed to do that task, and a small handful of companies like Nvidia collect all the profit that used to go to people like you paid to do that task.
You don’t own a data center and your labor is on an express train to being worthless.
So, fine, I agree with you. There’s a big payoff. You’re just not the one getting it.
I think this probably says more about you than the "AI"