>you took a single report that agreed with your statistics
These are not my statistics. I'm not affiliated with Faros at all. I built an analysis on top of their reporting.
And, it's also not one report. DORA has tracked statistics with respect to throughput and quality as well. Those indicators are flat for throughput and negative for quality. The throughput flatness is also supported by the shovelware data.
I discuss both of those lines in How I'm thinking: https://unessays.substack.com/p/how-im-thinking-about-the-va...
>you suggest that net value is lost simply because there are more incidents. this is a big jump
I don't think it's a big jump at all. Incidents and bugs drive rework. Rework has to be subtracted from throughput. Product throughput is the only thing people pay for.
This type of analysis is done all the time in manufacturing and devops. Here's a link for you: https://reworkcost.com/benchmarks. I'm not bringing novel intellectual ideas to the table here.
Faros reports a 16% throughput improvement on PRs. They also report an 860% code churn increase. If you assign only 9% of that increase to wasteful rework, then the absolute throughput improvement disappears. This is a very simple, straightforward analysis of the operations data reported by Faros.
> - you say that historically different technological improvements may have had similar patterns but this specific one is different because AI is stochastic
I'm saying LLMs are unreliable. I think we agree on that front, you say:
>I agree AI is stochastic and I'll put it this way: it is a high variance bet but it pays off.
What I'm disputing is the "pays off" statement. That statement is amenable to validation with data. In my view, the data is saying it doesn't pay off. I think it says that very clearly. Across distinct lines of evidence.
>if you are so sure this won't lead to enterprise level productivity, how do you think this will show in macro trends? Surely you must believe that the valuations must drop wouldn't you? Can you come up with a concrete future scenario that would vindicate your opinion that AI doesn't make enterprises more productive?
I think LLMs can deliver value in the enterprise. I think the way to do that is to use them as quality checks and not as primary authors of intellectual work - like writing code.
Unfortunately, this use case would not support the expected 2-10x productivity increases that current valuations depend on. I do expect a major market correction in the near future. It would not surprise me if OpenAI or Anthropic are acquired. I think we're at risk of that happening within the next 1-7 months.
What would invalidate my beliefs?
1. Actual micro or macroeconomic data indicating economic productivity is increasing.
2. A Faros like observational study demonstrating sustained throughput improvement with significantly less rework and quality impacts.
I think I could be swayed against the market correction if the financials of OpenAI or Anthropic are strong. I'm anticipating they will be quite bad. I think Mythos was very expensive to train and I think the improvements in capability are sublinear. The inference costs are incredibly high.
I also have ideas about how Anthropic and OpenAI are trying to change their business models into enterprise transformation plays. Similar to Palantir. But this comment is already long.
>If they don't build it, someone else might do it.
No other players in the market other than US tech companies have the capital or the technology to train the models of the power of Fable. The way the Chinese model builders are building their models is by distilling from US models. So Anthropic, by building Mythos with all this bio data, has created the possibility that other actors can distill their models and do harm with them. (Not to say the Chinese are seeking to build weapons, but actors with their models might).