Is it all just vapourware?
kirahowe.com
kirahowe.com
I think its fascinating just how much of a gap there is between what's being claimed, and the verifiable observable data of the open source world. Major open source projects are by and large starting to ban LLMs now, because the contributions made by LLM users have been universally terrible and unhelpful. There doesn't appear to be a single major project that's found generating code to lead to major productivity speedups, and the consensus appears to be that its just lead to a lot of crappy contributions that are harder to spot immediately as being obvious crap
I regularly see people claim that they are now 10x more productive with LLM code generation, and I just wonder where all the code is. Is it somehow true that these gains are only being realised in proprietary projects, and not a single one of them has put even a small fraction of their new found engineering powers into eg Godot? Why do only the poor quality LLM code generation users make PRs to open source projects, and never the engineers that know how to really use it correctly?
If you look in the open source major project space, you can find almost no evidence that AI code generation exists at all. Go browse your favourite critical tool and look for AI generated PRs that have landed in the codebase, its probably a tiny handful of them in comparison to the human written PRs prior to an LLM ban. It turns out that once you have a verifiable, open quality review bar, for some reason almost no LLM commits really meet the level of quality necessary
I strongly suspect that what we're seeing is that much of the tech code-writing economy had already become completely performative prior to AI turning up. It no longer matters in the current age if your code is good, or works, because your job is to give the illusion of product development while the stock market price gets pumped, until you all cash out your share value, get bought, or hop jobs in 2 years. For many companies it literally does not matter if you produce anything that generates value (or works), because the illusion of progress is all that matters. AI is absolutely incredible at creating the illusion of progress, because it looks a whole lot like real code, it just appears to have failed the bar of making actual projects that work. If that was never the goal in the first place, it probably really is a 10x productivity boost
I suspect the ban of AI in established projects to be a very complex decision. Even if core developers would like to use AI, they don't want to review all of the AI-generated code from the larger community. So the only consistent way to preserve sanity is to declare that nobody would use it. Good luck enforcing it though!
Nothing's stopping core developers from adopting LLM generated code for themselves, while banning it for external contributors
And if they're doing that, why would they tell you, or anyone?
And if they're doing that responsibly - collaborating with AI, and reviewing the code - they don't even have an ethical reason to tell about AI involvement, any more than telling about the StackOverflow answers or blog posts they read before coming up with some implementation.
Then all it'd take is one of these engineers blowing the lid on their whole secret LLM contribution strategy to absolutely break the industry wide open to the benefits of these massive productivity increases, and finally prove all the doubters wrong. They sure are good at keeping secrets that would directly benefit them to expose
So 0<whatever<1
Weeks for each feature in a greenfield project... Yeah, LLMs make unworking code way faster than that.
It could be that the engineers who are extremely productive with LLMs are landing PRs that look indistinguishable from good, hand-written PRs.
LLMs are superhuman at short term coding such as debugging and writing tests though and you're missing out by not using them there.
Commercial and industrial electricians are LLMs now?
Instead, software is plodding along exactly the same as it did prior to LLM code generation, and there's no evidence of superprogrammers making superprojects in 1/10th of the time. With a 10x productivity gain, what used to take a year should take a month
But that doesn’t mean they stick.
And the core issue isn’t AI, it’s that… a successful, highly adopted project requires time for people to know it exists, adopt it, and also time by the maintainer growing community and ensuring reliability. The people factor is more important. I see tons of new projects, but I’m not going to pick a brand new project that has high odds of being abandoned. I’m going to pick the one that consistently maintains it and has some adoption already.
I have seen engineers create very successful internal projects fairly quickly. And yes, a project that would have taken a year taking a month. And offering a lot of extra bells and whistles that you just wouldn’t have time for.
But these are greenfield internal projects, and the bar is much, much lower for those.
I have seen multiple internal incidents root-caused by an agent faster than the humans responding. Just because it can go up and down rabbit holes a lot faster.
The problem is that AI solves one bottleneck, but not others. One team member produces a huge amount of new PRs. (Like 12 solid enhancements and big fixes in a couple hours.) Now I have less time because I’m reviewing that. And we’re all context switching a lot more.
On top of that, I find AI workflows continue to be deeply immature, even though certain models are very effective and very good at troubleshooting.
The story around testing is not really improving for example. AI can write tests, but are they good? I don’t think we have much actually ensuring product quality and reliability automatically. Unit tests are not enough.
Collaboration is very poor too. Coworkers agent creates PR, and now I’m reviewing it, and now he sends my comments back to his agents… really clunky workflow especially since I’m ACTUALLY just prompting his agent.
Plus, certain models (Opus) are getting much worse at writing. I will not use Opus any more because the writing style is so horrible.
The constant change means approaches that worked well a couple months ago don’t work well today. And there are no real experts, because no one’s been doing this for long. And half the posts and learning out there are outdated, or straight up blogspam.
This makes it hard for people to learn and get better, despite the fact that models like Sol 5.6 are effective troubleshooters, and write decent code.
It seems likely that the gains from generating tons of code are being offset by the debt incurred to understanding what you're doing. We see lots of greenfield projects one-shotted with GPT or GLM or whatnot, but very little on the side of projects with long-term maintenance goals. This is telling, to me, that the _effective_ gains are much lower than perceived (it's lots of fun to see the thing crank out code at breakneck pace, probably contributing to this). Still quite nice, and very useful, but not a totally new paradigm.
Wherever these “10000x” engineering gains are… I’m wondering why they’re only for shit nobody sees or touches.
Where are all the killer new things? All we see are glorified hobby projects (guilty as charged) and an unimpressive pace of equally unimpressive feature additions to all the same old stuff.
Businesses with resources out there should tackling huge things we all want, that they never could have undertaken. But I’m not seeing it. Everything is just… more or less the same. And this tooling didn’t arrive last Wednesday.
Or did we get magical AI at a time when we’re also completely out of ideas? That seems unlikely.
This shit just isn’t adding up. We’re not seeing anything that isn’t more “here’s some more ai tools and services… they’re basically the same as the ones you saw last month.”
What for? Someone show me.
I agree they are over-hyped. Maybe the problem is that all the things LLMs are great for are essentially quality of life improvements. There are a dozen killer apps worth of functionality I get out of LLMs - near instant proofreader, one-shot code bugfixer, excellent cursory research tool, bibtext generator, etc. All of these things extend what I can do a bit but none of them instantly result in me being able to generate category killing apps. They still make stupid mistakes, they don't reduce my workload, but they reduce cognitive load on mundane tasks.
My guess is that open source folks overindex on being anti-AI.
Eg if you check out Godot, initially they were cool with LLM code, and then had to clamp down on it because it was not leading to anything good
For instance, we had one person submit a merge request to convert the entire GIMP codebase into Python, and it was just tons of files with stubcode implementations. Another person submitted a fix for our OpenEXR plug-in to load YCbCr images, and when I reviewed it, they'd "imported" a non-existent version of the library that had an imaginary "OpenYCbCrOpenEXR ()" function. :)
It's not just productivity, it's life changing.
AI lowers the bar to making submissions which permits flawed subissions to be thoughtlessly submitted by people who don't really understand what they are doing, technically or culturally.
It does, however, enable people who put in the effort to work on something significant. Those people are fewer in number, but exist. They do get to see the animosity that unwitting novices receive.
I don't think it would be surprising for those people to opt out of engaging with a toxic environment.
You might have something there.
It probably holds true for a lot of things. Either that or most gutairs sold are defective.
Or, failing that, you'd think any of these engineers - given the truly vast amount of programming power now available at their fingertips with their 10x productivity - would have simply replaced these projects. It'd only take a few 10xers who wanted to show that everyone else was wrong. That's how open source often works, someone just gets a been in their bonnet and then we get linux
Instead: I can't find any of it. There's no evidence of this productivity boost in the wild. There aren't new high quality 10x open source projects springing up that are replacing everything. There aren't high quality LLM contributions, feature development isn't going faster. There's almost no evidence of high quality AI code generation at all in the open source space, existing projects or otherwise
Where's all the code? I want the receipts if people are claiming a 10x productivity. Because at the moment, the much more likely explanation seems to be that its simply not true
For any instances where people have made things and revealed them to the world have been declared to have been grossly flawed by being held to a standard of scrutiny that no person usually gets.
It seems everyone is Cardinal Richelieu now.
What's the point of writing great software if nobody else is going to use it, and how do we even confirm there are these alleged secret geniuses hiding away somewhere? Surely their work would leak out by some nth-order effect.
I made a claim that absence of evidence is not evidence of absence.
That is not a claim of existence.
- pola-rs/polars: https://github.com/pola-rs/polars/pull/26823 - ~2.68x median speedup of primitive-to-boolean casting credited to Claude Opus 4.6
- pydantic/monty: https://github.com/pydantic/monty/pull/643 - ~53x speedup (488ms down to 9.2ms) of bytes substring search generated with Claude Code
- numpy/numpy: https://github.com/numpy/numpy/pull/31573 - ~21x speedup of Python datetime → datetime64 conversion, with Claude Code used for profiling and implementing the performance improvements
- apache/datafusion: https://github.com/apache/datafusion/pull/21182 - up to 49x speedup of LIMIT queries by eliminating unnecessary sorts and pushing limits into file scans, generated with Claude Code
- apache/datafusion: https://github.com/apache/datafusion/pull/21651 - ~4.39x speedup of ClickBench Q6 by resolving MIN/MAX directly from Parquet metadata instead of scanning columns, generated with Claude Code
- pola-rs/polars: https://github.com/pola-rs/polars/pull/27958 - ~3.2x speedup of Int8 Series sum, developed with assistance from Claude Fable and Claude Opus 4.8
- numpy/numpy: https://github.com/numpy/numpy/pull/31274 - up to ~1.44x speedup of common small NumPy reductions, with the fast path written by Claude Code and manually refined
Because if it was true then open source projects such as GIMP could basically be as feature heavy as Photoshop overnight.
LLMs amplify existing expertise. Give them to experts and you can get fantastic results. Give them to amateurs and you might get the occasional impressive demo, but you're not going to get anything that a responsible software team would commit to maintaining in the long term.
Assisted-by: Codex:gpt-5.5
Assisted-by: Claude:claude-opus-4.8
repo:torvalds/linux assisted-by (claude OR codex)
... but if you clone the git repo you can get an exact count of 824 commits (not sure why the search over-counted).
Here's a good candidate for a material performance improvement:
https://github.com/torvalds/linux/commit/e1bf79628453e6afac8...
Single-stream throughput (MB/s):
Before After Change
seq-write/dontcache 298 897 +201%
rand-write/dontcache 131 236 +80%There are two incentives for contributing to open source. The first is to make the app better (add a feature, improve the code, fix a bug, etc.) It's possible that LLMs don't meet the bar but if a human has put the effort in it's not always obvious that it's AI. More likely LLM code is accepted when it's good and rejected when it's bad. The system works.
However, the second incentive is that open source contributions are seen as a 'ahortcut' to making a name for yourself. Being a contributor on a big project goes on resumes and GitHub profiles. Often people who use AI for that don't review the code or even check it does what they say it does. That slop needs to be banned. The downside is losing the good contributions, but it's still a net win.
Selection bias? OSS has a thing proprietary projects don't - an endless cohort of opportunists trying to wedge in a "contribution" for personal gain, be it a Hacktober t-shirt or resume boost or an occasional vulnerability (with the resume boosting being by far most likely). The good LLM-based contributions, you probably won't recognize as AI-assisted unless the author explicitly decided to label them, and if they're really good and use LLMs responsibly, they probably don't even have any ethical obligation to label LLM involvement, much less any benefit.
OSS involvement in general stopped being an indicator of skill once Github activity became a factor for job applications.
Edit:
There's also a second factor: many people (myself included) use LLMs to code one-off personal tools with no intent of them having more than one user, ever. Get an itch, have an LLM scratch it for you, carry on with your life. You'll see little trace of it, because it's hard to distinguish a single-user program that's good enough for specific purpose from a pile of autogenerated slop that probably doesn't work, and neither of these will show up as PRs to big OSS repos anyway.
Instead, if you check out something like ImGui, we can see that its largely just Omar as usual with a bunch of PRs. Nothing much seems to have changed, feature development carries on at the same pace as it did previously. This is replicated across nearly every single major long term open source project I can find
The only reason developers now feel the need to hide their LLM usage is because the LLM contributions have all been shockingly bad, so you're reversing cause and effect. If LLM contributions had been incredible, they wouldn't be frowned upon socially now. Surely someone would blow the lid on this huge covert conspiracy about how incredible their contributions to.. SFML have been?
This effect also wouldn't prevent LLMgineers from creating their own open source projects that should be absolutely outstripping the existing open source space. After all how old is ImGui? With a 10x productivity gain, it should take a year of development to easily match its features if those gains are truly real now
>if they're really good and use LLMs responsibly, they probably don't even have any ethical obligation to label LLM involvement
The copyright issues means that non disclosure is inherently unethical
You're dismissing scale of things. OSS communities were having high-profile discussions about gatekeeping and discrimination way before LLMs were a thing, because already at that point the amount of human generated slop was becoming untenable. And yet, no one argued that this must be evidence that all human contributions are bad and humanity is a bubble waiting to pop.
> ImGui
Weird choice of an example. ImGui is common infrastructure. Who in their right mind would want to mess with that? Especially when building something to solve some end-user problem (whether N=1 or N=1 billion), ImGui is a component you include, not one you rewrite.
There's hardly any reason anyone has to contribute to that project, with or without LLMs.
Perhaps the biggest issue hasn't been the speed of development all along but people working on wrong things, hence why progress isn't as evident. LLM helping to build something that isn't needed faster isn't going to productivity more visible.
I wrote a game I left in a folder from 11 years ago in 2 months, front and back plus all the tooling, plus features I never would have attempted before. I recreated open-sourced projects from github, written in a language I wasn't familiar with, so I could better work with the application and include it in my backend codebase.
I rewrote my mail server in less than 2 days, which took me over 2 weeks to configure and setup the last time.
My life is easier, because the demands are the same but I have more free time. I research more, I learn more, and I try more things. Just like any software project, some are abandoned (even with LLMs) when running into unforeseen issues or mismatches with theoretical plans. Most importantly, I can make things that I think are exciting, with very little effort. The first 35 years of development, I struggled to be good enough to tackle sourcing various projects without becoming frustrated and mostly knowing myself well enough to avoid trying at all.
Now I can see myself writing passion projects until I'm dead.
According to this list of top Github repos by stars[1], of the top 10, nine of them are informational and one is OpenClaw, which certainly has plenty of AI generated code.
The next 10 include react which has a CLAUDE.md and a .claude, and lots of landed PRs that look like AI. And linux, which, we all read Linus' stance. He sees the value. And superpowers, ECC, and hermes-agent. All of which are AI stuff.
I think it would be reasonable to ask which major open source projects aren't using AI assistance?
> Major open source projects are by and large starting to ban LLMs now, because the contributions made by LLM users have been universally terrible and unhelpful.
That is not why. Or at least not the majority of it. The biggest part is that maintainers just can't keep up. They don't have time to do in depth reviews for the number of PRs coming in to find out if it: works, does what it says it does, meets a need, has docs, is well written, meets criteria, etc. That does not mean all those PRs are trash. Some might be trash, some might be gold. Just like human generated ones.
What it comes down to is vibes. Anti-AI folk don't like AI for reasons (that are valid) and they don't want to find any value in it. So they don't. But lots of people are. Lots of really cool, interesting, clever, new software is being written and getting used and building communities but people see CLAUDE.md and go "slop, no thanks" and won't even give it a try.
[1]: https://github.com/EvanLi/Github-Ranking/blob/master/Top100/...
Where are all the vast quantity of LLM commits for these open source projects, if these engineers are truly 10xing their productivity with LLM code generation? Because I regularly browse the source of many of these, and can find almost no evidence that LLM code generation is anywhere near the level of productivity being claimed for:
1. ImGui
2. SFML
3. SDL
4. GLFW
5. OpenSSL
6. Linux
7. GCC
8. Clang/LLVM
9. Libc++/libstdc++/msstl
10. Rust
11. Godot
12. Nlohmann
13. Boost
14. Python
15. GNU tools
Etc etc. There are even more major projects, but these are just the critical ones I can think of off the top of my head. If you go for a browse though eg Godot, you might think that LLMs don't exist. In fact if you go for a browse through any of these, you might think that LLMs are borderline never used for code generation in the open source world. This is a diverse collection of random projects, and yet for some reason none of these have experienced a 10x increase in productivity in any form from any contributors from LLM code generation
>That is not why. Or at least not the majority of it
Its not the quantity, if maintainers were receiving incredibly high quality PRs that could just be merged, they'd be very happy. Godot is an example of a project that was initially pro LLM, and then had to about face because the PRs were just absolutely crap
>Some might be trash, some might be gold
Is there any evidence that any of these PRs have ever been gold?
I think this is probably the answer. Working as a professional software engineer actually has very little to do with writing good code and maybe never has. The job is to “ship products”, and as long as you keep up a sufficiently convincing illusion of progress toward that end the bills get paid and nobody really cares about the impact on end users.
I'm coming from the perspective of infra management so that's my bias, and a very complex code base may make my perspective off mark.
But nowhere does it seem it to be a good idea for the whole thing to be LLM generated. It seems to view a big holistic problem in steps in isolation, causing it to add complexity on top of complexity as it's going through it's later stages. For example it'll create infrastructure designs that add layers of unnecessary and redundant tooling and business logic, when a human would have re evaluated step 1's path when it realized step 3 was doing the same thing again in a more complex way.
Two years ago, I couldn't just roughly describe my backlog and then have the code fixed. I had to type it out myself, run it, look at logs, fix toolchain issues, and so on. It was tedious. Or I could get a junior to do it.
Now can get these things done quite fast, without concentrating nearly as hard.
Clearly, it isn't vapour.
It delivers something. That something we have yet to figure out the best way to use, but there's definitely something there that works.
I get the feeling a lot of people are frustrated because the little gains are lost in organisational chaos, rather than the tools not working.
That's the result of AI adoption in a company being people just getting a subscription and doing their own thing, instead of systematically introducing and finding a common ground; a trivial example of that would be open source projects increasingly adopting LLM policies to get everyone on the same page.
I don't recommend web work, the last ten years or the next ten. Wouldn't touch the stuff. I skipped the phone app era, too.
It also wrote some unit tests that validated the kernel sampling weights, and wrote some Jupyter notebooks to go along with the kernel algorithms as comparisons.
It's not just web dev... It helps (a lot in some cases) if you ask very specific things rather than just "make this vague thing", but I'm more and more coming round to the conclusion it is now a useful dev tool (until two months ago I was a sceptic).
Then it probably would have taken me at least half a day to think about what tests to write and to make them.
It did it in around 5 minutes. I had to check it, and it wasn't perfect, but it compiled, ran, and was very close.
It's the time compression I'm impressed with.
So an unimportant, personal project. People seem to extrapolate being able to do something cool into being able to do useful work, which is what this whole discussion is about.
I'm also using Github Copilot at work, but I can't as easily talk about what I'm doing there.
People keep repeating your sentiment here but I simply can't follow, are we even on the same planet? Or did everyone switch to just not caring about maintainability and code quality anymore? Or are your work tasks simply so mindnumbingly, stupidly simple that even an AI can oneshot them properly?
I mean this honestly btw, not dismissively like some sister comments. The gap between the productivity increases people report on HN and what I experience myself is insane. In fact, if I factor in the procrastination I find myself doing on dotting the i's on a supposedly "one-shotted" AI implementation of a nontrivial feature, I think the AI actively slows me down.
The only way I've found that I can actually use AI productively and sustainably is in very small tight loops and, well, at that point it's not that much faster than just typing in the code (with the occasional "Cursor Tab" complete).
Am I doing something wrong?
If AI is as good as it being marketed as, you shouldn't even be able to hold it wrong!
“No true Scotsman” would require me to redefine “senior developer” to exclude people who disagree with me. I didn’t.
And “you’re holding it wrong” is a strange objection to a tool that, like Git, rewards learning how to use it. The question is whether that learning pays off.
A big problem is that these codebases rot. Agents move incredibly fast at first, but then as you pay less attention (or perhaps no attention at all) to the architecture, they slowly fall to bits. So then you decide, I'll use AI to rewrite it! And it gets better for a while until, well, you get it.
That's not to say there isn't value here, there absolutely is, just — chill. A little.
Find some code you're proud of. Find someone else's code you admire. Find something that's been stable for 20 years with no bugs reported. Feed it in, ask it to look it over, and prepare to be humbled.
Where things go off the rail is when you want it to plan AND implement features. The blind spots of LLMs are not where they are for humans and way more work to anticipate. You have to stay on top of the bucking bronco, but you CAN move much faster if you can architect your system so more tasks fall in the "obvious" bucket - that is where the art of engineering still lives. Human understanding remains the goal.
LLMs will certainly be an aide, but assuming adoption of them is required across a whole _field_ ignores whole classes of problems, domains, and constraints the production of software covers.
Rather, we should be wary of allowing our skills and hard-earned knowledge to atrophy by over reliance on new technology that is far from perfect, reliable, or even universally available. These tools more than any before suffer from the junk-in-junk-out problem and I’d much rather work with someone who knows their fundamentals than someone who knows their way around a the LLM of the month.
They aren't saying LLM adoption is required across 100% of the field. They are pointing out that LLMs have reached an impressively capable state, and someone insufficiently inclined to test new tooling that they still dismiss LLMs as "just fancy autocomplete" is exactly who gets left behind when technology advances.
Two things can be true at the same time, 1) that LLMs are not required across a whole field, and 2) that software engineers unwilling to acknowledge their utility have as much a place in the future as the software engineers of 50 years ago who refused to use a compiler.
If they're so easy to use, wouldn't the opposite be true? i.e. people who over rely on LLMs become weaker at the core competency? Nobody gives a rat's ass if you did something in an hour or took all night. The deadline is still next week because of all the meetings. What's the point? Anyone at any point in human history can tell you that doing too much too early has extremely high odds of failure. You're much better off thinking about the business concerns at hand instead of getting lost in the weeds playing with the code.
There's also zero chance of the cadence speeding up because software engineers were never the bottleneck, and LLMs don't work so well for everyone else's job.
If you can't pull rabbits out of hats when the time is right without an LLM, you're already at a disadvantage compared to everyone else who can (anyone who isn't a junior dev today).
What if what people become weaker at, is no longer the core competency? Then people who maintain their skills at what used to be the core competency, at the expense of what is actually the core competency, will be left behind.
The rest of your comment, re: nothing taking less than a week because of meetings and software never being the bottleneck, is not something generally true across the industry. I'm sorry that that has been your experience; it does not sound pleasant.
You're correct that the focus is no longer just writing code after a few years of experience, but it doesn't change the fact that being held accountable for the code means it is one of your core competencies.
You cannot seriously believe anyone is going to throw the baby out with the bathwater by giving away their power to "AI". Why else do you think a dev is in all those meetings? Why do they get paid to "hurry up and wait"? That's not inefficiency. That's the real work.
Those who find this reality unpleasant have always been the ones to switch careers (and then realize all careers are the same). I'm not sure what you're trying to say other than you'd like to muddy waters because you really like AI and are allergic to code and accountability. Do not be so naive.
Piss-poor management by scleratic and top-heavy organizations, mainly.
You're trying to start a fight by casually accusing me of stances I didn't take, mixed in with insults. There are healthier ways to deal with your insecurities.
Nothing I said is incompatible with devs being responsible for what they produce. Though you seem to be saying that someone should be fired because their code managed to hit a nondeterministic compiler bug? Why didn't they double check the assembly? They should take accountability.
Edit: although you might be subject to an NDA... But this is pretty much my test for "AIs will take all the jobs": can it write truly safety-critical software yet?
Most programmers are not experienced enough to write such code themselves off-hand, but more importantly, the safety-critical systems are made safe by following a strict process, not by skills of individuals, which makes it orthogonal to involvement of LLMs.
Also, weird choice of example. 99.9% of coding is not safety-critical, so whatever reservations this would imply (even if it actually doesn't imply any), don't apply anyway.
Why? This has already done via existing processes. My point was to illustrate that if the future is LLMs then surely these processes wouldn't be needed anymore? After all, the LLM would just... Do it itself.
> ... the safety-critical systems are made safe by following a strict process, not by skills of individuals, which makes it orthogonal to involvement of LLMs.
And an LLM that is going to take all the jobs wouldn't be able to execute that process independently and with little oversight?
If my example is, to you, not a good one, what would you rather I use? Most of the common ones can be overly trivialized/minimized (particularly by someone who is uninterested in admitting that LLMs can't do something). That is not to imply that the gp is this kind of individual, but far too many people who I ask to do this (or something similar) are exactly that kind of person: believing that LLMs are insanely great and can't admit (or see) the cons.
I do agree that the current pace of abandonware creation is wild. I think a lot of projects are generated top down from an idea, and not bottom up through usage. If you have a new experimental project, and you don't have recent commits, I assume you are not using it and thus it has no value.
I think agentic use is mostly only valuable as author extension (help you search, rubber duck, generate code but needs very heavy review). Over time with enough usage of a process I do think some of them can turn into author automation but not from simply writing down an idea. It takes lots and lots of executions, iterations, generalizations, specifications, basically lots of work to get any decently hands-off intelligent automation through agents.
The problem is really that the promise was so big but the management and implementation was so painful, coupled with non-stop updates, coupled with far too big of an implementation surface.
A classic "lets solve 15 problems at once" situation, but none of them really well.
I'm Matt, I run Product & Engineering at Ona.
First, thank you for trying the product and for taking the time to write this up. I shared the post with our product engineering team. There are several things in your experience that simply aren't good enough, and we're working on them. I've also credited $200 to your account in case you do want to explore further.
To be concrete:
> My very first encounter with the app was that I was unable to login on their desktop version at all. Auth is hard, so I can empathize and forgive this.
Whilst I appreciate the forgiveness, we hold ourselves to a higher bar than this.
We've done extensive testing of the desktop authentication flow and haven't yet been able to reproduce the failure you experienced. If you're willing, could you email me at matt at ona dot com? I'd like to grab some logs and work out what went wrong.
> it spent nearly my entire $20 worth of “ona compute units”, whatever those are, thrashing and trying to get a hold of the todos from linear just so it could pick one to start.
Looking through what happened, there were a few different things going on here.
- Roughly a quarter of the OCUs were spent by our devcontainer setup agent. That agent created a PR which standardises the development environment and adds install, build and CI tasks so that future agents can operate in a reproducible environment. There is real value in doing that setup once, but we did a poor job of making it obvious that it was happening, why it was happening, and what you were paying for. We'll fix this. We've also switched that setup agent from Sol to Luna as of today after tuning it against our evals. This should make that setup substantially cheaper going forward. - The Linear flow also involved far too much friction. You went through multiple authentication and setup steps before the agent could actually get to the work you wanted it to do. That's not the experience we want. We have shipped a fix that lowers this friction already, and have a couple more planned (that will take a little longer). - OCUs themselves are our attempt to combine model and compute consumption into a single unit. If someone has paid us money and still can't tell what they're spending it on, that's a problem. We're actively revisiting how we explain and expose this.
> This only adds more friction between me and my projects, which is literally the opposite of what I want when I pay for developer tooling.
I agree completely. Today, Ona asks a lot of a new user before it has earned the right to ask for that investment: connect this integration, authenticate that service, let us configure the environment, understand what an OCU is.
The thing we need to get better at is making the initial experience more boring: sign in, point Ona at something useful, and see it accomplish something valuable before you have to think about any of the machinery underneath.
> The thing is I think ONA is a good idea. I am evidently willing to pay for this kind of tooling. But I want a version that works without lighting twenty dollar bills on fire.
Based on your experience, I can see why you reached this conclusion.
If you're open to it, I'd love to spend an hour with you to help you get Ona working on something useful and show we can achieve what we outline in our marketing. No expectation that it changes your opinion but I'd like the opportunity to learn from what went wrong and show you what the product should have felt like the first time. Email is as above.
I look forward to hopefully connecting soon.
- Matt
I still think of what I'm doing as software engineering, and I'm glad that I had many years of professional and hobby development before using agents since I think that's given me the ability to make good architectural decisions (and helps me resteer the LLMs when they want to do something suboptimal), but my involvement in actually writing code is quickly going to zero. That said, they aren't perfect and they still introduce bugs, but I believe the quality of my current product is higher than what I would have created pre-agentic coding.
Things I've found helpful in keeping quality high:
- Visual regression tests (detect UI bugs before you commit them)
- Fuzz testing of interfaces and app behavior
- Automatically add regression tests for any bug that I/the LLM fixes
- Logging/alerting that tracks an errors/invariant violations triggered in the app
- Performance metrics that are surfaced in a dashboard.
All of these are very easy to add since the LLM can create this infrastructure for you. The fuzz testing in particular is something very few products I've previously worked on have since most people don't know how to implement it. I ran the fuzzers for a few minutes and they quickly caught multiple subtle bugs that I was not aware of.
This is a real product that helps a real, non-VC funded service business, and although I could have made something similar myself it would have taken me a lot longer, be harder to use, and probably be less reliable.
Edit: while it's true that you can quickly blow through the $20/month plan, the $200/month plan allows you to get a lot done and is basically sufficient for my needs. It's also very cheap when you consider what it would cost to pay someone to do similar work.
I have worked on other projects where flaky tests are a problem (and generally cause developers to ignore and submit anyways), but so far I've avoided that. My experience with flaky tests is that they're typically due to poor modularity or to subsystems that other teams can modify. This project exists in a monorepo and I'm the sole developer so that's not a problem here.
This is one of the reasons that I've simply not bothered with a lot of these types of AI products. It feels like gambling. Maybe I'll spend $20 on tokens and end up with something awesome. Or maybe I'll spend $20 on tokens and end up with nothing useful and then I'll be glad it was only $20 I lost.
(I wouldn't suggest basing my evaluation of the entire field of coding agents around that particular product.)
There are definitely guys in some companies just pointing an agent swarm at 50 Github tickets and saying "go get 'em boys" and hence need a lot of orchestration tools but for the rest of us do we really need the AI to do all the build steps and the testing steps... or do we just need the new feature that runs when the button is clicked and then we can check it out in the browser ourselves. The code gen is what I need the AI for, not the 'smoke tests' and the tsc finagling. I'm willing to spend some mental calories myself in figuring this feature out step by step
Actually this makes me wonder if some people are just not very into building step by step like "okay first let's check the JSON shape the endpoint returns", "okay now let's make a simple form", etc
I still don‘t know if this is real. If it works like that, where are the companies that only kept every tenth, 10x‘d AI enhanced dev? They should have significantly higher margins. The companies that keep all devs and 10x‘d them, finally bringing that backlog to 0 should have exploding revenue and profit.
Where does AI create value? I can‘t see it.
I use it every day, but nothing got faster.
Why fire anyone in this scenario? Growth is only going to keep coming.
I don't think devs are that expensive relative to a big company.
Airbnbs revenue last year was 12b. If they have 1000 engineers,maybe average 200k/yr that's only 200m. If they got rid of their engineers still only a fraction of of the revenue
That's accurate. There are a lot of people (some of them quite senior and/or skilled) who operate in the way you described.
There's no such thing as a free lunch, and anyone trying to build software without significant guardrails and insights into the process will have to give up control of their codebase.
Ray Myers does a good job of exploring this topic in a recent Software Should Work talk, trying to answer: "Is code for people or AI?" [0]
We've seen similar waves with new technologies before: overexposing the pros, dismissing the cons, hyper-optimism, and people using a lot of jargon without saying much of substance. The difference this time is the scale of the impact and the volume around it.
May I fix that for you?
The product is being launched before the value is there.
That's like saying smart watches are useless based on trying out the smart watch made by will.i.am's tech company rather than Apple's.
I can absolutely see where this technology works; but like EVERYTHING THAT THESE VC FUNDED FASCISTS IGNORE, just because it works in the lab does not mean it works in real life. I can see exactly where this technology fails: it lets you spool out so much thread into the abyss that you're just going to keep refactoring everything, all the time and when it blows a hole in your project, whelllp! you wanted to refactor that too. Imagine having 100 side projects grossly built and still, you just want to move on.
Fiishing projects are still going to need work and the explosion of "this is _my_ engineering harness" is everyone twiddling their thumbs cause real projects of significant value are still about scope and "total addressable market" and not "I made an AI to diddle children, so you dont have to" crowd.
That statement somehow amazes me. Obviously, LLMs need plenty of steering, but I don't get how a company can be built on that.
You see a version of this in every tech boom cycle, but this one is way more reflexive.
Mature technical software projects are slow moving because they have real users building other real software and businesses with them. Their engineering involves navigating the ecosystem and dep and compatibility constraints.
Any project moving super fast pumping out PRs either isn’t burdened/blessed with real usage OR doesn’t care about (reap real value from) the users it has.
Related theory is that the LLM “labs” are actually not particularly incentivized to prioritize stable API surfaces. They want machine interfaces to constantly rapidly change so that we must depend on model based natural language surfaces to navigate the complexity.
God this times 1000. I'm so sick of having AI chase a bug and having to lead it by the hand like a toddler to try and help me solve bugs. The difference is, a toddler learns when you explain. An LLM holds your responses in context and uses them to generate more authentic sounding garbage, but next session, foomf, the lesson is unlearned again.
I had a tiny problem with prerendering not working on a react site last night, I must have spent close to an hour running commands it asked me to run like a moron, verifying dumb things I had already checked like "did the file actually upload to ftp" or had I restarted Apache or was there some magic htaccess file somewhere. Eventually I just gave up on Claude and tried some different paths in Apache virtual host settings and it worked.
Of course if I went back to Claude and reported that, as would be my instinct with a human, it would confidently explain why that was the bug, why it couldn't spot the issue, and how smart I am for finding it. And then it would forget everything tomorrow
This hasn't been my experience at all. I can give it a vague description of the problem and have it find it pretty easily without any more input from me.
I do think there's a pretty big variation in tools and setups and what output people are getting. I'm now just using OpenAI Codex in VS Code and it churns through problems like they're nothing.
Sometimes I have to get it to not over-engineer a solution; not because it's necessarily wrong but because I don't need that much correctness.
Next stop will be Perl: write only language.
And then we will hit the "Java EE" wall. Again.
And our LinkedIn will get red hot from recruiters looking for "experienced crisis devlopers".
Better read https://blog.senko.net/code-was-never-the-hard-part-is-an-in...
I almost wonder if there's a campaign to start suppressing public awareness of where SOTA capabilities really are.
That Nvidia finances a lot of this should give anyone with a clue on finance pause, that it doesn’t tells you you’re in the middle of a bubble.
It's nothing short of magic. And in terms of productivity, features that took months are now being done in days. I'm not even sure what the author is talking about frankly.
Am I missing something?
Again, are we still on this ridiculous concept. I stopped reading at that sentence.
Is that going to be the revisionist future that the industry tells itself - ""one day, LLMs came along and suddenly computers could program themselves, and it was an absolute revolution, but it made no difference at all to anything because "The hard part of software engineering was never writing the code." so people stopped using LLMs because of that. And that's how LLM's came to an end after being a short trend.""
The second part was that the test was being administered by a gatekeeper who was hired before those practices were put in place.
The 3rd was what? Amazon or Netflix burning your ass out because they are anti-human companies?
But, it could be even quicker if I just give up and willingly write shitty code that works. I don't do that, but I could, and I doubt anyone outside of close developers on my team would ever notice.
No. Anyone making this claim is being disingenuous.