AI usage patterns in software teams | Hacker News Reader

AI usage patterns in software teams

linear.app

This looks like measuring what is easy to do, rather than what really matters.

PR open counts, issues created , ceos/founders spending more time on linear don't automatically lead to better outcomes (in my experience they are often negatively correlated:-) )

This looks like measuring what is easy to do, rather than what really matters.

Even that's hard. There aren't enough signals to attribute changes directly to AI, so these apps seem to correlate the signal that the user was interacting with AI to the changes they made e.g "Bob used AI at that time, and they opened a PR at a similar time, so Bob probably used AI to make that PR."

Until the tooling for gathering data on AI usage improves the data will be fairly interesting because correlations often point to something related, but won't be a source of truth.

I'd be interested to overlay, I don't know, customer satisfaction or anything that can show the follow-on effect of all this output. Linear won't have that information.

My guess is some will jump up (where the team has managed to make themselves move effective and responsive) and many will plummet (doesn't need explaining).

Then there might be something to look at.

I'd be interested to overlay, I don't know, customer satisfaction or anything that can show the follow-on effect of all this output. Linear won't have that information.

Very few businesses can accurately attribute customer value to the work they do, especially once they're passed start-up scale. A mature company makes lots of small changes and they're rarely measurable.

Yeah all of these along with raw token usage are metrics that were being used around December to February by people who just didn't have anything to go off yet.

Skill/Hook usage rates, budget spend, auto-approval rate, focus area heatmaps, MTTR, MTTD are all there now

Usage does not correlate with valued output or ROI. This article smells like a bad attempt to say hey guys our customers see ROI without any actual evidence that they do.

Also, somewhat amusingly, the “founder” consistently being at the top of the use curve might just have something to do with everyone else also using it, but that more implies people are using it because the chief at the top wants them to use it… not because it’s actually useful. A pattern that’s typical of bad AI deployments.

> Usage does not correlate with valued output or ROI

This sounds possible but how can we know? It could just as well correlate. Effort of all kinds correlates with success even when it's not obvious or not... linear.

If that was true then there wouldn’t be a giant ROI crisis fueling calls of an AI bubble burst imminent. If the answer was “the ROI is clear” then folks would just say that.

The debate about ROI isn't about whether everyone is going to throw their hands up and stop using AI tooling - the debate is about where that value is going to be captured. It's pretty clear that using some degree AI assistance for programming is on the whole faster and better than writing code by hand for the majority of applications, and it's clear that the degree to which it's better than writing code by hand is increasing quickly.

I'm sure someone is going to reply to and say "actually it's not clear and the tech debt is about to explode everything real soon now" or something to that effect, but it's hard for me to explain this theoretically when it is so obvious in practice in my day to day.

ROI doesn’t mean “does it help.” Few question if it helps.

ROI is about how much its costs vs the value of the “help.” That’s where the present crisis is. Is the ROI there to pay for the trillions in commitments that have been bet on that ROI? That’s a clear no at this point hence the growing panic.

> Is the ROI there to pay for the trillions in commitments that have been bet on that ROI? That looks like a clear no at this point.

That's only a potential crisis for those who have made concrete investments.

On the use side, the 'cost' of AI spans more than two orders of magnitude. Looking at recent models (<6mo) with reasonable performance (intelligence index >= 45) on OpenRouter, the output cost ranges from $50/MTok (Fable) to $0.153/MTok (DeepSeek Flash 0731).

From the perspective of a user of LLM/agent assistance, there's very likely a range where the benefits outweigh the costs.

If the ROI for the model developers isn't there, then that just impairs the future trajectory of the field. Current models are just bits that aren't going anywhere, and as long as they can be served (in inference) above their marginal cost they will continue to be so-delivered.

On the value capture question, it's really not clear that the ability to write code at superhuman speed actually translates to an increase in the speed with which we can create new, better technology products that customers will pay more money for, vs. just enabling layoffs.

You could say the same thing about compilers versus assemblers, high-level languages versus low-level ones, and services and libraries versus monolithic programs.

All other things being equal, increasing the speed of some part of the development process will increase the overall pace of development. However, By Amdahl's law that increase will be sublinear, and that is why we should take "pull requests" as an imperfect metric.

We also don't get to pick the form that 'better technology products' take. While we'd probably like to keep cost(/effort) and complexity constant and increase robustness and performance, the market equilibrium might be 'worse is better' and reward whiz-bang features and lower effort.

Yep.

Also, large companies are still experimenting on how to integrate LLMs into their workflows. Due to the fast cadence of releases, people forget that LLMs became robust (regardless of the capability level/parameter count/data size) enough to use semi-reliably in company-specific ways only 1 year ago. And bigger the org, the slower the process. I don't expect it to settle and get productive used across the majority of very large companies for another year atleast.

High prices are mostly a result of DC capacity. As more and more DCs get built out, prices will drop. At a unit level the inference business is extremely sound regardless.

> You could say the same thing about compilers versus assemblers, high-level languages versus low-level ones, and services and libraries versus monolithic programs.

Right, but we didn't invest a trillion dollars of capital into any of those things in the span of a couple years, thus forcing them to capture value and show a return on such a massive investment.

These days, my work has become: generate code for 20 minutes, then spend an hour reading it.

And then more hours cleaning it up and re-prompting.

I program as a hobby, personal projects because I can.

I recently set up a local llm to see what the fuss is about and other than the few ringer solutions my experience is as you described. 2min promping, 5min waiting, 3hrs debugging or just doing it myself.

I am very likely doing it wrong, and it does speed up some aspects, but I wouldn't say I trust llm code any more than my own. Until it runs and throws an error, the llm is 100% confident that it has written perfect code.

Models that run on (average) consumer hardware are not even close to comparable to models like Fable or Sol. Its like comparing an ant to the largest dinosaur.

You don’t even have to go that far, recent Opus models are quite impressive.

A normal agentic loop will have the agent using a type system and basic tests to do some basic validation of changes. A good agentic loop would give the agent a very easy way to verify if it’s on the right track. I think agents are better than many humans at writing error free code (runtime errors, not bugs. The code could still be buggy or incorrect.)

Fundamentally though, an agent cannot produce great code, because great code requires intent, which is the opposite of the statistical mean.

You will get a solution that works with a proper workflow, but you won't get one that scales or would be truly maintainable. Which is also what you get with random midwit drive-by contributors, but faster. I'll give it that.

I'm not sure I agree that LLMs solve for the statistical mean

But more importantly: isn't what you call "intent" just a series of optimization goals? You want your code to satisfy the constraint of being correct™, while also maximizing various other goals like being maintainable, easy to understand, having few lines of code, as little tight coupling as possible, etc. Goals that often conflict, but when given two implementations you could likely tell which hits the better tradeoff (in your engineering experience)

Those are all things that theoretically - with a tight enough specification and enough compute - a constraint solver could solve. No human intent necessary.

The issue is more that we can't fully specify all those side goals, and even if we could the LLM would struggle following them. A classic paperclip maximizer problem (where nobody told the paperclip maximizer to keep the planet inhabitable and all the other side conditions we implicitly assume)

> I'm not sure I agree that LLMs solve for the statistical mean

The good thing is that you don't have to agree with that, as the fundamental technical reality does it for you. LLMs do work like that. They are just statistics and probabilities.

Doesn't the universal approximation theorem apply to typical LLM architectures? LLMs solve for the shape of the input data, as well as they can approximate it in the constraints of their architecture, the optimizer and the amount of compute spent in the training process

And where those constraints prevent us from matching the shape there is no guarantee that we match anything like the mean. Though maybe we can agree that that's where the optimizer would tend to steer towards when it can't do any better

If you built a task management system, you'd have very different code bases depending on whether it's for internal use at a mid-size development org or as a SaaS.

So I wonder whether, in your experience, the results you've seen, could have improved by providing sufficient context? - or what context was given.

I.e. if you have the agent that same context, as one of your colleagues would have/require to solve a problem.

"You're holding it wrong"

You've missed my point. I didn't dismiss agents. I did dismiss the industry.

I don't need to add more context to a statement that operates on a layer above where context injection would influence it. It is a conceptual impossibility. Not a technical roadblock.

This is a weird framing, but the user is the "intent". If you can produce "great code" by yourself (whatever that is), then you can get your agent to produce it for you. If you are talking about vibe-coding then yeah, it is going to be trash

> because great code requires intent

If can put properly engineered intent in the prompt that is verifiable, it works wonders. Anything that can defaults to the llm doing its way, you're right, it just can't converge to good, not with proper constraints.

That has been my experience as well. Having the agent have full access to what it is building feels like a cheat code. A recent example: Godot (the game engine) has a feature where GPUParticles can spawn another particle system when they collide with something. Today (as in, download the engine right now and see it for yourself) this is broken, particles spawn beneath the collision box and the faster the particle is the worse it is.

So I asked Claude to first figure out a way to instrument GPU particle code so it could itself check his solutions. After that I left the agent running for 3 hours and it came back with a solution (continuous collision detection) and a bug fix (particles get their velocity applied an extra frame after colliding). I'm sure someone would find something to complain about the code (which is why I haven't upstreamed it) but it looks visually perfect and I had plan to fork the engine for my project anyway

Once agents run autonomously for longer periods preventing vision drift becomes as important as correctness.

Epiq solves this with an architecture that supports workflow auditing, allowing you to time-travel state in a filtered view to reconstruct what happened, when, by who, and where intent started drifting, while also allowing you to correlate the evolution of the board with the corresponding commits.

https://ljtn.github.io/epiq

I don't think setting up a local LLM is a reasonable way to get a good idea of how enterprises are using this stuff.

Try using Fable and report back. Local LLM is to Fable as Little Tike car is to a Porsche.

Local llms aren’t super exciting unfortunately.

Neither are proprietary blackbox systems unfortunately.

Felt that. I already gave DeepSeek the HTML and it still said the UI was good to go, then told me it has no vision.

Like the Titan submarine team's moto mine is - real men test in production.

The LLM produce so much code that the best I can is skim and look for obvious flaws, also pass it trough adversarial one.

I spend 45 minutes writing it. then i git commit and move on cuz i made something good.

who is winning here? lol

this seems inappropriate. I think its a bad paradigm that just because you use a platform's service, they get intimate details about your usage. And for them to be so bold about publishing the statistics they've stolen from their customers data? Gives me a reason to never recommend my org use this platform.

The data is aggregated, and you cannot possibly identify a single user from what's been published. I see no issue here.

I think their point is that there's business value in the usage data and linear is using that value in a way that benefits them but not the customers they got it from.

It reminds me of matt levine's reframing of insider trading where it's not about fairness it's about theft. You're supposed to get secret insights and use them to get an edge. What you can't do is get an edge for yourself with secret insights that your employer got.

So it roughly comes down to "that data is valuable and rightfully belongs to the originating company." Which then makes this a contract diligence type situation.

My company uses Linear. The data presented in this blog post is worthless to me and the company. It can be crudely summarized as “agents and agentic development processes are conducting more Linear operations.” Duh!

Whether you make a good trade with the stolen information is spurious though.

I don't believe this very strongly. Again it's a contract issue not a moral one. But the metaphor still holds I think.

https://archive.is/rVwku - Matt Levine writing about insider trading

lol isn't this what google said and yet..the NSA cometh

I’d rather it was published for free in a blog post than sold without my knowledge. As a linear user. This shit doesn’t matter, everyone in the industry know these kinds of stats are being tracked, I’m happy for a company to be transparent about it.

Why would a company want to leak its processes and workflows to another company in a capitalist system. Seems sloppy and a short sighted transfer of wealth to external stockowners.

Uhh that's how a lot of economic data works too. Guess how we get a lot of jobs data... ADP

Oh hey! My industry. Guess what? ADP doesnt just yoink your data. ADP conducts voluntary surveys on the scale of hundreds of thousands. Voluntary. ADP also pays for it for the most part. Do you think Linear conducted..voluntary surveys here?

"Pull requests are up 111% in two years". Would be more honest to say that the number of pull requests "detected" by linear are up XXX%. Because it only works if you setup git repo tracking and use it properly. And at that point it is not obvious if more teams are using linear and using it correctly, or if the number of PR really increased that much!

Doubling PRs isn't necessarily unbelievable. AI reduces ceremony around PRs, makes the small shitty changes that you never had time to do possible and the larger cleanups at least doable.

Double the PRs doesn't mean double the output.

This looks like a "We are so AI native and efficient!!" article. At least, they could delve deeper into how they define the metrics and how they collected the data.

> delve

Butt-head: Huh huh huh. Hey Beavis, he said "delve". Huh huh huh.

Beavis: Yeah, I bet he'll use an em-dash next. Heh heh heh.

This is pretty interesting but I wish it would have been refined in two ways:

- The prose before the data appears to be AI generated. Not a big deal but it makes the reader work harder to figure out what's actually being said.

- Linear didn't control for their platforms AI changes over the past year. The platform has become much more AI integrated, so a lot of numbers will move. I'm not sure how you do it but this is only useful signal with a control.

> Time spent on customer requests, docs, and projects held steady [..] AI has so far changed how teams execute far more than how they decide what to build

I think the measurement for this may be flawed. We do mostly use AI to decide how to build. But what we build is influenced by AI-driven research into a problem or task. That's largely done in coding and desktop AI tools, not Linear Asks/AI.

I'm working on accelerating my team's work by implementing AI-driven code pipelines with guardrails to eliminate as much unnecessary review time as possible. Also making a chatbot for turning repetitive tasks & PRs into buttons, and an "architectural guidance" chatbot that gives advice tailored to our business, software/system architecture, cloud, standards, etc. This puts AI and automated jobs in the center of both how (automated task) and what (architecture guidance).

But this has a not-so-great implication for Linear. With my tools, a human never has to touch a ticket, so we could use any ticketing system with an API or CLI. Linear is a great product because they made a great interface. What happens when I replace their interface with a chat bot?

I think it's worth noting that headlines like "AI adoption has spread to every function" is only limited to roles covered/tracked by Linear.

Recent studies by e.g. Google show a much broader range of adoption.

Nice data!

Tim (author) if you're there: it'd be amazing to see the split of which agents people are using, if you have that data.

Founders topping the usage chart says more about org pressure than product value. Adoption driven from the top looks different from adoption driven by results.

I didn't know Linear has "AI features". Linear is boring, but that's actually fine by me.

I use LLMs to write my code, but this does not show up in this data.

The AI Slopologists strike again. More garbage by garbage people.

Is this just an opinion? If so, fair.

If it's an attempt at rebutting their claims etc, it'd be easier to interact if you provided some data, or concrete observations :)

The bullshit asymmetry principle suggests that once something is identified as bullshit, we shouldn't spend much time to write an excruciatingly detailed refutation.

Invoking the bullshit card on opposing views, in order not to have to elaborate your own pov, seems counterproductive, if the aim of your participation in a discussion is to test your own hypotheses about the world - or in any way gain or deliver insights.

Can't tell if you mean the people the article is talking about or the article itself

¿Por qué no los dos?

I might build a Chrome extension that looks for “AI” in HN submission titles and shows a fake comment at the start saying “AI is all slop garbage that’s total slop and I hate it. AI sucks and it’s slop!”

It will save people time reading the 5-10 other identical vacuous comments

Reply on news.ycombinator.com