Google CEO says more than a quarter of the company's new code is created by AI
businessinsider.com
businessinsider.com
But the code completion engine is basically just good at finishing the lines I'm writing. If I'm writing "function getAc..." it's smart enough to complete to "function getActionHandler()", and maybe suggest the correct arguments and a decent jsdoc comment.
So basically, it's a helpful productivity tool but it's not doing any engineering at all. It's probably about as good, maybe slightly worse, than Copilot. (I haven't used it recently though.)
My productivity isn't so much enhanced. It's only 1%... 2%... 5%... globally, for each employee.
Have you ever dabbled with, mucked around in, a command line? Autocomplete functions there save millions of man-hour-typing-units per year. Something to think about.
A single employee, in a single task, for a single location may not equal much gained productivity, but companies now think on much larger scales than a single office location.
Work gets scheduled on short time frames. 5% savings isn't enough to change the schedule for any one person. At most, it gives me time to grab an extra coffee. I can't string together "foregone extra coffees" into "more tasks/days in the schedule".
Now making a kettle faster? That might actually be something.
Wait, no. That should be based on how much slacking off Google employees do ordinarily, an unknown quantity.
It’s probably closer to 10% than 100%, especially at big companies.
One thing I would love to see is reports of benefits from various tools coming with one’s typing ability in WPM. I’d also like to see that on posts where people express a preference for “a quick call” or stopping by your desk rather than posting what they want in chat. I have some hypotheses I’d like to test out.
I regularly describe it as "The best snippet tool I've ever used (because it plays horseshoes)".
I would be very confused if someone told me that they uncritically used the generated code from a snippet program with no manual input or understanding, and I feel the same with Copilot. At best, it suggests an auto-complete that I read and interpret before accepting.
The closest I come to "code generation" is during test writing, where occasionally I will let the description generate some setup, but only in tests where there are a broad number of examples to follow, and I am still going to end up re-writing a decent chunk of it based on personal example. I would not "let it write the test suite for me" and then trust the green, and I suspect that would easily fail code review (though it would be an interesting experiment...).
Obviously your comment as a good goof and well made, but it does speak to a little bit of the disconnect between what is being touted as an "AI coding tool" and how I, a person who makes react native apps to pay my rent, actually use the dang thing (i.e., "A pretty good snippets plugin"). Is My code 'AI generated'? I wouldn't call it that, but who can say definitively? We're in a fun new semantic world now.
Use OpenAI to convert a few thousand lines of code from a language you're familiar with to one you’re not, as all the state-of-the-art tools in the field above use that language. Debug all the issues that arise from the impedance mismatch between the languages. Recreate the results from the seminal paper in the field to verify that the code works, and run it on your own problem. Write a stream-of-consciousness post without spell-checking, then throw it into GPT and ask it to fix it.
I want the end of the line completed with focus on context from the working code base, and I don't want an entire 5 line function completed with incomplete requirements.
It is really impressive when it implements a 5 line function correctly, but its like hitting the lottery
(By "unread" I meant that I don't look very closely before deleting if it looks weird.)
And then write tests. Or perhaps I wrote the test first.
I think it’d be more useful if it was clipboard aware though. Sometimes I’ll copy a type, then add a param of that type to a function, and it won’t have the clipboard context to suggest the param I’m trying to add.
if err != nil {
return fmt.Errorf("Cannot open settings: %w", err);
}"classic" intellisense is reliable, so why introduce random source in the process?
Swift is especially frustrating because it will hallucinate the method name and/or the argument names (since you often have to specify the argument names when calling a method).
Or when I say I need to do something, it invents a library that conveniently happens to just do that thing and writes code to import and use it. Except there's no such library of course.
In IntelliJ thankfully you can disable that part of the AI, and keep the part that you trigger it when you want something from it.
This is a fantastic description of how it disturbs my coding practice which I hadn't been able to put into words. It's like someone is constantly interrupting you with small suggestions whether you want them or not.
Realtime tab completion is good at some really mundane things within the current file.
You still need a chat model, like Claude 3.5 to do more explorational things.
When I copy and paste code, very often it needs some small changes (like changing all xs to ys and at the same time widths to heights).
It's very good at this, and does the right thing the vast majority of the time.
It's also good with test code. Test code is supposed to be explicit, and not very abstracted (so someone only mildly familiar with a codebase that's looking at a failing test can at least figure the cause). This means it's full of boilerplate, and a smart code generator can help fill that in.
Then it got worse a couple of years ago when they tried some early-stage AI approach. I turned it off. I expect that next time I update VS it'll have got substantially worse and it will have removed the option for me to disable it.
If so, all your code is sent to cloud.
An extreme example of this would be the AWS GovCloud for government/military applications.
Would I send the source of a trading algo or chatgpt to a third party, probably not but those are the outliers. The code for your xyz SAAS does not matter.
I am probably an outlier in that I don't really care what corpus a LLM trains off of. Its its available in the public space, go for it.
https://www.techspot.com/news/104945-ai-coding-assistants-do...
I agree with your take though, it does seem helpful to juniors but not beyond that (yet), and this OP stat seems dubious unless juniors are doing a big portion of the work.
It is not as good with questions about API documentation for popular java libraries though and it will just hallucinate APIs/method names.
If I ask it a generic question like "how can I create a class in Java to invoke this API and store the data in this database" it is pretty useless. I'm sure I could spend more time giving it a better prompt but at that point I can just write the code myself.
Overall they are a better search engine for stackoverflow, but the LLMs are not really helping me code 30% faster or whatever the latest claim is.
Code is often a liability.
I feel it's a bit like the old "measuring developer productivity in LoC" metric.
As I hinted at in another comment, in Java if you had a "private String name;" then the following:
/**
* Returns the name.
* @return The name.
*/
public String getName() {
return this.name;
}
I am going to argue contrary. If AI increases productivity 2x, it opens up as much new usecases that previously didn't seem worthy to do for its cost. So overall there will just be more work.
This is the entire history of the computing industry. We’ve been automating our work away for decades and it just creates more demand.
Well I do freelancing as well besides my usual day to day work, and that's also where direct benefits apply, and I'm getting more and more work, overwhelmingly so.
Which is how they've surpassed 25% in new code, as compared to the 10% (made up number, but clearly non-zero) in the past. But incremental improvement, is all.
They allow me to do much more than that thanks to all the knowledge they contain.
For instance, yesterday I wanted to write a tool that transfers any large file that is still being appended to to multiple remote hosts, with a fast throughput.
By asking Claude for help I obtained exactly what I want in under two hours.
I'm no C/C++ expert yet I have now a functional program using libtorrent and libfuse.
By using libfuse my program creates a continuously growing list of virtual files (chunks of the big file).
A torrent is created to transfer the chunks to remote hosts.
Each chunk is added to the torrent as it appears on the file system thanks to the BEP46 mutable torrent feature in libtorrent.
On each receving host, the program rebuilds the large file by appending new chunks as soon as they are downloaded through the torrent.
Now I can transfer a 25GB file (and growing) to 15 hosts as it is being written too.
Before LLM this would have taken me at least four days as I did not know those libraries.
LLMs aren't just parrots or tab completers, they actually contain a lot of useful knowledge and they're very good at explaining it clearly.
Did you use it in your editor or via the chat interface in the browser? Because they are two different approaches, and the one in the editor is mostly a (pretty awesome) tab completion.
When I tell an LLM to "create a script which does ..." I won't be doing this in the editor, even if copilot does have the chat interface. I'll be doing this in the browser because there I have a proper chat topic to which I can get back later, or review it.
But using copilot as a better autocomplete is really helpful and well worth the subscription. Just while typing as well as giving it more precise instructions via comments.
It's like a little helper in the editor, while the ChatGPT/Claude in the browser are more like "thinking machines" which can generate really usable code.
However i think that you might open source the thing with a disclaimer of no maintenance. Whoever is willing to maintain it can just fork it and move along.
But it's not a production quality implementation of new need.
I mean if you assume all devs are script kiddies who simply copy paste what they find on google (or ChatGPT without asking for explanations) then yeah it's never gonna be useful in a prod setting.
Also you're very wrong to believe every technical need or combination of libraries has already been implemented in open source before.
Moreover Claude can explain the functions used very clearly (if you're too lazy to jump to definition in your editor)
LLMs are becoming actually useful to developers new to a language. Just as Google was 20 years ago.
I have shipped production code using LLMs in languages I did not study approved by seasoned SWE's is evidence that an acceleration is happening.
This is what's problematic with modern "AI". Most people inexperienced with it, like the parent commenter will uncritically assume these LLMs poses "knowledge". This I find the most dangerous and prevalent assumption. Most people are oblivious to the fact how bad LLMs are.
People misusing tools don't make tools useless or bad. Especially since LLMs designers never claimed the compressed information inside models is spotless or 100% accurate, or based on logical reasoning.
Any serious engineer with a modicum of knowledge about neural networks knows what can or can't be done with the output.
The general public has a very different idea of that though and I frequently meet people very surprised the entire profession hasn’t been automated yet based on headlines like this.
Well no, but we have no evidence it can be used for the whole stack, whatever that means.
Asking for evidence is not being an ostrich.
But I doubt the predictions from men whose net worth depends on the hype they foment.
In the hands of a junior, AI can create a false sense of confidence and it acts as a technical debt and security flaw multiplier.
We should bring back the title "Software engineer" instead of "Software developer." Many people from other engineering professions look down on software engineers as "Not real engineers" but that's because they have the same perspective on coding as typical management types have. They think all code is equal, it's unavoidable spaghetti. They think software design and architecture doesn't matter.
The problems a software engineer faces when building a software system are the same kinds of problems that a mechanical or electrical engineer faces when building any engine or system. It's about weighing up trade-offs and making a large number of nuanced technical decisions to ultimately meet operational requirements in the most efficient, cost-effective way possible.
For instance, one I find very useful is that we have this pattern of checking the result of a function call, logging the error and returning, or whatever. So now, every time you have `result = foo()`, it will auto suggest `if (!result) log_error...` with a generally very good error message.
Very basic, but damn convenient. The more patterns you use, the more helpful it becomes.
It is actually a testament that, part of Google's code are ... kinda formulaic to some degree. Prior to the LLM take over, we already heard praise how Google's code search works wonder in helping its engineer writing code, LLM just brought that experience to next level.
This workflow is not perfect but I am definitely building out all the core features way faster than if I wrote the code myself, and the code is in quite a good state. Quite often I do some bits of cleanup, refactorings, making sure typings are complete myself, then update ChatGPT with what the code now looks like.
I think what people miss is there are dozens of different ways to apply AI to your day-to-day as a software engineer. It also helps with thinking things through, architecture, describing best practices.
- it only works well when you write code from scratch, context length is too short to be really helpful for working on existing codebase.
- the output code is pretty much always broken in some way, and you need to be accustomed to doing code reviews to use them effectively. If you trust the output and had to debug it later it would be a painfully slow process.
Also, I didn't really noticed a significant difference in code quality, even the best model (GPT-4) write code that doesn't work, and I find it much more efficient to use open models on Groq due to the really fast inference. Looking at ChatGPT slowly typing is really annoying (I didn't test o1 and I have no interest in doing so because of its very low throughput).
This is kind of true, my approach is I spend a fairly large amount of time copy-pasting code from relevant modules back and forth into ChatGPT so it has enough context to make the correct changes. Most changes I need to make don't need more than 2-3 modules though.
> the output code is pretty much always broken in some way, and you need to be accustomed to doing code reviews to use them effectively.
I think this really depends on what you're building. Making a CRM is a very well trodden path so I think that helps? But even when it came to asking ChatGPT to design and implement a flexible data model it did a very good job. Most of the code it's written has worked well. I'd say maybe 60-70% of the code it writes I don't have to touch at all.
The slow typing is definitely a hindrance! Sometimes when it's a big change I lose focus and alt-tab away, like I used to do when building large C++ codebases or waiting for big test suites to run. So that aspect saps productivity. Conversely though I don't want to use a faster model that might give me inferior results.
...yet. Bugs can take time to surface.
It can work, but what a terrible developer experience.
> I'd say maybe 60-70% of the code it writes I don't have to touch at all
I used to to write web apps so the ratio was even higher I'd say (maybe 80/90% of the code didn't need any modification) but the app itself wouldn't work at all if I didn't make those 10% changes. And you really need to read 100% of the code because you won't know upfront where those 10% will be.
> The slow typing is definitely a hindrance! Sometimes when it's a big change I lose focus and alt-tab away, like I used to do when building large C++ codebases or waiting for big test suites to run.
Yeah exactly, it's xkcd 303 but with “IA processing the response” instead of “compiling”. Having instant response was a game changer for me in terms of focus hence productivity.
> I don't want to use a faster model that might give me inferior results
As I said earlier, I didn't really feel the difference in quality so the switch was without drawbacks.
Interesting, personally I have noticed a difference. Mostly in how well the models pick up small details and context. Although I do have to agree that the open Llama models are generally fairly serviceable.
Recently I have tended to lean towards Claude Sonnet 3.5 as it seems slightly better. Although that does differ per language as well.
As far as them being slow, I haven't really noticed a difference. I use them mostly through the API with open webui and the answers come quick enough.
Sometimes that results in code, but it's the research and cross referencing that's actually useful with it
There is this tool Aider. Takes your prompt, adds code files (sometimes not all of your code files but files it figures relevant) and prepares one long prompt, sends it to an LLM, receives the response, and makes a git commit based on the response. If you rather review git commits, it can save you the back-and-forth copy-pasting. https://aider.chat/
There's not even anything wrong with that, don't take my comment the wrong way. It is an interesting question of what happens at scale though. We could easily find ourselves in a spot where very few people know how to code and most producing code don't actually know how it works and couldn't find or fix a bug if they needed to. It also means LLMs would be stuck with today's code for a training set until it can invent its own coding paradigms and languages, at which point we're all left in the dust trusting it to work right.
1. This quote is clearly meant to exaggerate reality, and they are likely including things like fully automated CL/PR's which have been around for a decade as "AI generated".
2. I stated before that if a team of 10 is equally as productive as a team of 8 utilizing things like copilot, it's fair to say "AI replaced 2 engineers", in my opinion. More importantly, Tech leaders would be making this claim if it were true. Copilot and it's clones have been around long enough know for the evidence to be in, and no one is stating "we've replaced X% of our workforce with AI" - therefor my claim is (by 'denying the consequent'), using copilot does not materially accelerate development.
That's only worth doing if you're trying to cut costs though. If the company has unmet ambitions there's no reason to shrink the headcount from 10 to 8 and have the same amount of output when you can keep 10 people and have the output of 12 by leveraging AI.
Note the difft between "cost cutting" (do less, to lower cost) and "efficiency" (do same, but with less cost)
If your engineers become 20% more efficient then your margins are better and your problem is solved. (Indeed if you have tech that can make any engineer 20% more efficient then you are back in the game of hiring as many as you can find, as long as each added engineer brings in enough additional revenue.)
Even if that's been happening, I don't think it would be politically savvy to admit it.
In today's social climate claiming to replace humans with AI would attract the wrong kind of attention from politicians (during an election year) and from the public in general.
This would be even more unwise to admit for a company like Google who's an "AI producer". They may leave such a language for closed meetings with potential customers during sales pitches though.
Don't think the public will be that concerned about people in Google's salary bracket losing their jobs.
For example they thought that twitter had a bloated workforce because of videos like this (https://www.youtube.com/watch?v=buF4hB5_rFs).
And a lot of people heavily disagreed with how they handled moderation. You can take things like the hunter Biden laptop suppression or in the funny category you had the getting banned for saying learn to code (https://reason.com/2019/03/11/learn-to-code-twitter-harassme...).
Take random company without controversies and you will find less vitriol about them getting fired.
Depends on who you ask.
If Trump wins and Elon Musk actually gets a new job, they would be bragging about replacing humans with AI all day long. And corporates are going to love it.
Not sure about what voters think though. But the fact that most of these companies are in California, New York etc means that it barely matters.
A new economy is forming and there is nothing that can stop it without causing major, unintended fallout.
In a similar vein, solving world hunger is closer today than it's ever been. The previous best hope was global thermonuclear war, but honestly that would leave enough survivors as to be mostly ineffective, and much more likely to have the opposite result. Severe climate change has a better shot at fully eliminating [human] hunger.
Has either bragged about this at all?
The only thing I've heard floated is Musk running a "government efficiency commission" which I just assumed meant he would be looking for ways to gut a lot of the never ending, never dying government programs. I've never heard him saying the commissions goal was to replace people with AI.
https://www.newsnationnow.com/politics/2024-election/trump-m...
The former president said such an audit would be to combat waste and fraud and suggested it could save trillions for the economy.
As the first order of business, Trump said that this commission will develop an action plan to eliminate fraud and improper payments within six months.
Would it be? Do they care?
Sam Altman's been talking about how GenAI could break capitalism (maybe not the exact quote, but something similar), and these companies have been pushing out GenAI products that could obviously and easily be used to fake photographic or video evidence of things that have occurred in the real world. Elon's obsessed with making an AI that's trained to be a 20-year-old male edgelord from the sewer pits of the internet.
Compared to those things, "we've replaced X% of our workforce with AI" is absolutely anodyne.
Altman encourages anyone that will listen to him that monopolies are the only path to success in business. He has a lot riding on making sure everyone is addicted to AI and that he’s the one selling the shovels.
Google isn’t far off.
Most capitalists have this fantasy that they can reduce their labour expenses with AI and continue stock buy-backs and ever-increasing executive payouts.
What sucks is that they rely on class divisions so that people don’t feel bad when the “overpaid” software developers get replaced. Problem is that software developers are also part of the proletariat and creating these artificial class divisions is breaking up the ability to organize.
It’s not AI replacing jobs, it’s capital holders. AI is just the smoke and mirrors.
When I was working in RPA (robotic process automation) about 7 years ago, we were explicitly told not to say "You can reduce your team size by having use develop an automation that handles what they're doing!"
Even back then we were told to talk about how RPA (and by proxy AI) empowers your team to focus on the really important things. Automation just reduces the friction to getting things done. Instead of doing 4 hours of mindless data input or moving folders from one place to the other, automation gives you back those four hours so your team can do something sufficiently more important and focus on the bigger picture stuff.
Some teams loved the idea. Other leaders were skeptical and never adopted it. I spent the majority of those three years trying to selling them on this idea automation was good and very little time actually coding. Its interesting seeing the paradigm shift and seeing this stuff everywhere now.
As a non-politically savy person ;-) I have a feeling that this is a similarly dangerous message, since what prevents many teams to focus on really important things is often far too long meetings with managers and similar "important" stakeholders.
1. Almost every business has growing workload. That means reassigning good employees and not hiring new headcount, not firing existing headcount. Unipurpose, low-value offshore teams are the only ones who get cut (e.g. doing "{this} for every one of {these}" work).
2. Most operational automation is impossible to build well without deep process expertise from the SME currently performing it. If you fire that person immediately after automating their task, what do you think the next SME tells you, when you need their help?
Successfully scaling operational automation programs therefore rely on additional headcount avoidance (aka improving their volume:employee ratio) and value measurement (FTE-equivalent time savings) to justify/measure.
AI engineers will not yet get a Nobel prize for putting everyone out of work.
Most likely what is actually happening is that the X% of workforce you would lay off is being put to other projects and Google in general can take on X% more projects for the same labor $$. So there is no real reason to make that particular "replaced" statement.
I haven't seen this yet so I'm intrigued. Is this a commercial product, or internal tooling?
When you maintain an open source project on GitHub you will occasionally get some open source automated bot that submits a PR to do things like this without you even asking, and I’m sure there’s plenty more you can sign up for or implement yourself.
I wouldn’t really call it AI, but it is automated. I agree with the parent comment that a journalist trying to push an angle would probably lump it in as AI in order to make the number seem larger.
You don't want a single PR that does that, because that would affect thousands of projects, and if something goes wrong with a single one, the whole PR needs to be rolled back.
However in my experience the system is much more powerful than you described. Maybe this is because I'm mostly writing C++ for which there is a much bigger training corpus than JavaScript.
One thing the system is already pretty good at is writing entire short functions from a comment. The trick is not to write:
function getAc...
// This function smargls the bleurgh
// by flooming the trux.
function getAc...
Often when I don't know exactly what function / sequence of functions I need to achieve a particular outcome, I put in a comment describing what I want to do, and Copilot does the rest. I then remove the comment once I make sure that the generated code actually works.
I find it a lot less flow-breaking than stackoverflow or even asking an LLM.
It doesn't work all of the time, and sometimes you do have to Google still, but for the cases it does work for, it's pretty nice.
This is one that I really liked: https://www.reddit.com/r/ProgrammerHumor/comments/l5gg3t/thi...
We call it “Telephone” because “Chinese Whispers” not only sounds racist, it is also super confusing. You need a lot of cultural context to understand the particular way in which Chinese whispers would be different from any other set of whispers.
Implementation comments belong inside the implementation, so they should be over if not deleted.
Remember, LLMs are just compilers for programming languages that just so happen to have a lot of similarities with natural language. The code is not the comment. You still need to comment your code for humans.
When I'm maintaining other people's code, or my own after enough time has gone by, I'm very interested in that sort of comment. It gives me a chance to see if the code as written does what the comment says it was intended to do. It's not valuable for most of the code in a project, but is incredibly valuable for certain key parts.
You're right that comments about why things were done the way they were are the most valuable ones, but this kind of comment is in second place in my book.
Something like:
query = query.orderBy(field: "username", Ordering.DESC)
Time will tell whether it outputs worse, equal, or better quality than skilled humans, but I'd be very wary of anything it suggests beyond obvious boilerplate (like all the symbols needed in a for loop) or naming things (function name and comment autocompletes like the person above you described)
It isn't something I worry about at all. If it doesn't work and starts creating bugs and horrible code, the best places will adjust to that and it won't be used or will be used more judiciously.
I'll still review code like I always do and prevent bad code from making it into our repo. I don't see why it's my problem to worry about. Why is it yours?
Functional bugs in edge cases are annoying enough, and I seem to run into these regularly as a user, but there's yet another class of people creating edge cases for their own purposes. The nonchalant "if it doesn't work"... I don't know whether that confirms my suspicion that not all developers are aware of (as a first step; let alone control for) the risks
Edge cases will usually be the ones to get through. Most developers don’t correctly write tests that exercise the limits of each input (or indeed have time to both unit test every function that way, and integration test to be sure the bigger stories are correctly working). Nothing about ai assist changes any of this.
(If anybody starts doing significant fully unsupervised “ai” coding they would likely pay the price in extreme instability so I’m assuming here that humans still basically read/skim PRs the same as they always have)
This isn't a new concern. Thoughtless software development started a long time ago.
So that's not to say there is nothing to be concerned about on stackoverflow, just that the risk seems manageable and understood. You also nearly always have to fit it to your own situation anyway. With the custom solutions from generative models, this is all not yet established and you're not having to customise (look at) it further if it made a plausible-looking suggestion
Perhaps this way of coding ends up introducing fewer bugs. Time will tell, but we all know how many wrong answers these things generate in text as well as what they were trained on, giving grounds for worry—while also gathering experience, of course. I'm not saying to not use it at all. It's a balance and something to be aware of
I also can't say that I find it to be thoughtless when I look for answers on stackoverflow. Perhaps as a beginning coder, you might copy bigger bits? Or without knowing what it does? That's not my current experience, though
Copilot will autocomplete entire functions as well, sometimes without comments or even after just typing "f". It uses your previous edits as context and can assume what you're implementing pretty well.
0. https://github.com/google/closure-compiler/wiki/Annotating-J...
AI coding assistants are generally really good at ramping up a base level of tests which you can then direct to add more specific scenario's to.
I really mean no offense, but your example doesn't sound much different from what old IDEs (say, Netbeans) used to do 15 years ago.
I could design a Swing ui and it would generate the code and if I wanted to override a method it would generate a decent boilerplate boilerplate (a getter, like in your example) along with usual comments and definitely correct parameters list (with correct types).
Is this "AI Code" thing something that appears new because at some point we abandoned IDEs with very strong intellisense (etc) ?
I remember many years ago as a Java developer, Netbeans could do such things as complete `psvm` to "public static void main() {...}", or if you had a field "private String name;" you could press some key combination and it would generate you the getter and setter, complete with javadoc which was mandatory at that place because apparently you need "Returns the name.\n @return The name." on a method called getName() in case you wondered what it was for.
On the other hand, LLMs are completely different -- based on machine learning and everything is random and about statistics. It depends on training data and context. It is more useful but make a ton of mistakes.
I wonder if the enormous hype around AI is a good or bad thing; it's obviously both but will the good win out the bad, or will the disappointment eventually be so overwhelming as to extinguish any enthusiasm.
[1] https://www.reuters.com/legal/ai-generated-art-cannot-receiv...
It's easy to see that adding up quickly to represent large percentages of the codebase by line, but it's not feature development or solving hard problems.
And it works maybe... 80% of the way and I spend all my time fixing the remaining 20%. Anecdotally I don't "feel" like this really accelerates me or reduces the time it would take me to do the change if I just implemented the translation manually.
But yeah, I wish the new version of Chrome worked better. ¯\_(ツ)_/¯
All things that I would consider tedious housekeeping, but nothing that needs serious reasoning.
It's basically a glorified LSP.
1. We take a lot of care to make sure the AI recommendations are safe and have a high quality bar (regular monitoring, code provenance tracking, adversarial testing, and more).
2. We also do regular A/B tests and randomized control trials to ensure these features are improving SWE productivity and throughput.
3. We see similar efficiencies across all programming languages and frameworks used internally at Google and engineers across all tenure and experience cohorts show similar gain in productivity.
You can read more on our approach here:
https://research.google/blog/ai-in-software-engineering-at-g...
edit: typo
That is, I, personally, am not measured on how much AI generated code I create, and while the number is non-zero, I can't tell you what it is because I don't care and don't have any incentive to care. And I'm someone who is personally fairly bearish on the value of LLM-based codegen/autocomplete.
For teams you can measure meaningful outcomes and improve team metrics.
You shouldn’t really compare teams but it also is possible if you know what teams are doing.
If you are some disconnected manager that thinks he can make decisions or improvements reducing things to single numbers - yeah that’s not possible.
How? Which metrics?
Measuring it is not the hard part.
The hard part is doing anything about it. If you can't attribute specific outputs to specific inputs, you don't know how to change inputs to maximize outputs. That's what managers need to do, but of course they're often just guessing.
"You can [accurately and meaningfully measure software engineering productivity] - but not on the level of a single developer and you cannot use those measures to manage productivity of a specific dev."
At the level of a company like Google, it's easy: both inputs and outputs are measured in terms of money.
I am not Amazon person - but from my experience 2 pizza teams was what worked and I never implemented it myself just what I observed in wild.
Measuring Google in terms of money is also flawed, there is loads of BS hidden there and lots of people paying big companies more just because they are big companies.
But I do code myself, I write requirements so I do know which ones are trivial and which ones are not. I also see when there are complex migrations.
If you work in a group of people you will also get feedback - doesn't have to be snitching but still you get the feel who is a slacker in the group.
It is hard to quantify the output if you want to be removed from the group "give me a number" manager. If you actually do the work of a manager so you get the feel of the group like who is "Hermione Granger" nagging that others are slacking and disregard their opinion, you see who is the "silent doer" or you see who is "we should do it properly" bullshitter you can make a lot of meaningful adjustments.
So that's how animal husbandry came about!
Even that would be hard since hunting is complex. If you are the one chasing the pray into the arms of someone else, you surely want it to be considered a team effort.
You need like 'blueberries picked'.
Then I would have to go on outlining 6-12 months of trying stuff out.
Because if I just give "an example" I will get dozens of "smart ass" replies how this specific one did not work for them and I am stupid. Thanks but don't have time for that or for writing an essay that no one will read anyway and call me stupid or demand even more explanation. :)
This seems to be useful to understand and internalize that there are no simple answers like "use story points!".
There is also loads of people who don't understand that, so I stand by that is useful and important to repeat on every possible occasion.
I think it's harder to measure things like developer productivity. The closest thing we have is making an estimate and seeing how far off you are, but that doesn't account for hedging estimates or requirements suddenly changing. Changing requirements doesn't matter for DORA as it's just another sample to test for deployment.
Unfortunately there's a lot of lag
A great generalisation and understatement! Often looking like you are becoming more efficient is more important than actually being more efficient, e.g you need to impress investors. So you cut back on maintenance and other cost centres and the new management can blame you in 6 years time for it when you are far enough away from it to not hurt you.
None of this works to evaluate individuals or even teams. But it can be effective at evaluating tools.
To use your example, a user with an LLM might say "LLM please fix this" as a first line of action, drastically improving this metric, even if it ruins your overall productivity.
The superstar developer’s secret… he would send blank reports to clients (who would only realize it days later, and someone else would end up redoing the report), and he would score many more points without doing anything. I’ve seen this happen a lot in many different companies. As a friend of mine used to say, “it’s very rare, but it happens all the time.”
I have no doubt that AI can help developers, but I don’t trust the metrics of the CEO or people who work on AI, because they are too involved in the subject.
1) They can work to improve the system
2) They can distort the system
3) Or they can distort the data
ah, to be young again...
Now, I was really bad at capitalizing on it, so nothing much came of it, but still, there are some positive things that higher-ups do notice.
I’m far from impressed with the output of GPT/Claude, all they’ve done is weight against stack overflow - which is still low quality code relative to Google.
What is probability Google makes this a real product, or is it too likely to autocomplete trade secrets?
I don't think this is a bad thing - if this can be accompanied by an increase in software quality, which is possible. Right now its very hit and miss and everyone has examples of LLMs producing buggy or ridiculous code. But once the tooling improves to:
1. align produced code better to existing patterns and architecture 2. fix the feedback loop - with TDD, other LLM agents reviewing code, feeding in compile errors, letting other LLM agents interact with the produced code, etc.
Then we will definitely start seeing more and more code produced by LLMs. Don't look at the state of the art not, look at the direction of travel.
That’s a huge “if”, and by your own admission not what’s happening now.
> other LLM agents reviewing code, feeding in compile errors, letting other LLM agents interact with the produced code, etc.
What a stupid future. Machines which make errors being “corrected” by machines which make errors in a death spiral. An unbelievable waste of figurative and literal energy.
> Then we will definitely start seeing more and more code produced by LLMs.
We’re already there. And there’s a lot of bad code being pumped out. Which will in turn be fed back to the LLMs.
> Don't look at the state of the art not, look at the direction of travel.
That’s what leads to the eternal “in five years” which eventually sinks everyone’s trust.
Humans are machines which make errors. Somehow, we got to the moon. The suggestion that errors just mindlessly compound and that there is no way around it, is what's stupid.
It is by the juice of Sapho that thoughts acquire speed, the lips become stained, the stains become a warning...
that "hallucinate" term is a marketing gimmick to make it seem to the gullible that this "AI" (i.e. LLMs) can actually think, which is flat out BS.
as many others have said here on hn, those who stand to benefit a lot from this are the ones promoting this bullcrap idea (that they (LLMs) are intelligent).
greater fool theory.
picks and shovels.
etc.
In detective or murder novels, the cliche is "look for the woman".
https://en.m.wikipedia.org/wiki/Cherchez_la_femme
in this case, "follow the money" is the translation, i.e. who really benefits (the investors and founders, the few), as opposed to who is grandly proclaimed to be the beneficiary (us, the many).
from a search for grand vs grandiose:
When it comes to bigness, there's grand and then there's grandiose. Both words can be used to describe something impressive in size, scope, or effect, but while grand may lend its noun a bit of dignity (i.e., “we had a grand time”), grandiose often implies a whiff of pretension.
it's even more fundamental than that.
even if they had any model, they would not be able to think.
thinking requires consciousness. only humans and some animals have it. maybe plants too.
machines? no way, jose.
USA (s)election, I guess.
Even if we accept the premise (seeing humans as machines is literally dehumanising and a favourite argument of those who exploit them), not all machines are created equal. Would you use a bicycle to fill your taxes?
> Somehow, we got to the moon
Quite hand wavey. We didn’t get to the Moon by reading a bunch of text from the era then probabilistically joining word fragments, passing that around the same funnel a bunch of times, then blindly doing what came out, that’s for sure.
> The suggestion that errors just mindlessly compound and that there is no way around it
Is one that you made up, as that was not my argument.
We got to the moon using a large number of systems to a) avoid errors where possible and b) build in redundancies. Even an LLM knows this and knew what the statement meant:
https://chatgpt.com/share/6722e04f-0230-8002-8345-5d2eba2e7d...
Putting "corrected" in quotes and saying "death spiral" implies error compounding.
https://chatgpt.com/share/6722e19c-7f44-8002-8614-a560620b37...
These LLMs seem so smart.
Sure, I'm really poor painter, Midjourney is better than me. Are they better than a human trained for that task, on that task? That's the real question.
And I reckon the answer is currently no.
You’ve just proven my point. My issue with LLMs is precisely people turning off their brains and blindly taking them at face value, even arduously defending the answers in the face of contrary evidence.
If you’re basing your arguments on those answers then we don’t need to have this conversation. I have access to LLMs like everyone else, I don’t need to come to HN to speak with a robot.
Of course self-driving cars aren’t a nonsense idea. The execution and continued missed promises suck, but that doesn’t affect the idea. Claiming “humans are geniuses without equal” would be pretty dumb too, and is again something you’re making up. And something doesn’t have to be “all snake oil” to deserve specific criticism.
The world has nuance, learn to see it. It’s not all black and white and I’m not your enemy.
Actually understand LLMs in detail and you'll see it isn't some huge waste of time and energy to have LLMs correct outputs from LLMs.
Or, don't, and continue making silly, snarky comments about how stupid some sensible thing is, in a field you don't understand.
Yes, they do *seem* smart. My experience with a wide variety of LLM-based tools is that they are the industrialization of the Dunning-Kruger effect.
Now once you've recognized that, you're better equiped for task at hand - which is augmenting and ultimately automating away every task that humans-as-machines perform by building equivalent or better machine that performs said tasks at fraction of the cost!
People who want to exploit humans are the ones that oppose automation.
There's still long way to go, but now we've finally reached a point where some tasks that were very ellusive to automation are starting to show great promise of being automated, or atleast being greatly augmented.
If humans are machines, then so are fairies.
The conceit that humans are machines carries with it such powerful ideology: humans are for something, we are some kind of utility, not just things in themselves, like birds and rocks. How is it anything other than an affirmation of metaphysical/theological purpose to particularly humans? Why is it like that? This must be coming from a religious context, right?
I cannot at least see how you could believe this while sustaining a rational, scientific mind about nature, cosmology, etc. Which is fine! We can all believe things, just know you cant have your cake and eat it too. Namely, if anybody should believe in fairies around here, it should probably be you!
Because it's boring stuff, and most of us would prefer to be playing golf/tennis/hanging out with friends/painting/etc. If you look at the history of humanity, we've been automating the boring stuff since the start. We don't automate the stuff we like.
Recognizing that humans, just like birds are self-replicating biological machines is the most level-headed way of looking at it.
It is consistent with observations and there are no (apparent) contraditions.
The spritual beliefs are the ones with the fairies, binding of the soul, made of special substrate, beyond reason and understanding.
If you have desire to improve human condition (not everyone does) then the task at hand naturally arisies - eliminate forced labour, aging, disease, suffering, death, etc.
This all naturally leads to automation and transhumanism.
LLMs get their errors fed back into them and become more confident that their wrong code is right.
I'm not saying that's completely unsolvable, but that does seem to be how it works today.
Start adding different prompts, different models and you get all kinds of ways to catch errors. Just like humans.
No. LLMs can be told that there was an error and produce an alternative answer.
In fact LLMs can be told there was an error when there wasn't one and produce an alternative answer.
https://chatgpt.com/share/6722e41d-6b20-8002-8cbb-3012cd9179...
But a LLM is not "simply" doing anything. It's extremely complex and sophisticated. Once you go from tokens into high-dimensional embeddings... it seems these models (with enough training) figure out how all the concepts go together. I'd suggest reading the word2vec paper first, then think about how attention works. You'll come to the conclusion these things are likely to be able to beat humans at almost everything.
In my experience, if you confuse an LLM by deviating from the the "expected", then all the shims of logic seem to disappear, and it goes into hallucination mode.
ChatGPT here went with a lexigraphical order in Python for some reason, and then proceeded to make false statements from false observations, while also defying its own internal logic.
"six" > "ten" is true because "six" comes after "ten" alphabetically. "ten" > "seven" is false because "ten" comes before "seven" alphabetically.From what I understand of LLMs (which - I admit - is not very much), logical reasoning isn't a property of LLMs, unlike information retrieval. I'm sure this problem can be solved at some point, but a good solution would need development of many more kinds of inference and logic engines than there are today.
Really? That must be a very recent development, because so far this has been a reason for not using them at scale. And noone is.
Do you have a source?
There was a recent meme about asking LLMs to draw a wineglass full to the brim with wine.
Most really struggle with that instruction. No matter how much you ask them to correct themselves they can’t.
I’m sure they’ll get better with more input but what it reveals is that right now they definitely do not understand their own output.
I’ve seen no evidence that they are better with code than they are with images.
For instance, if the time to complete only scales with length of the token and not the complexity of its contents then it probably safe to assume it’s not being comprehended.
Are you sure you wanted to say that? Or is the other way around?
Indeed, and one of the most interesting errors some human machines are making is hallucinating false analogies.
It's an insanely conservative tool
Lastly, it’s also a huge waste of energy to feed the same information over and over again for each query.
- context over training is like someone referencing docs vs vaguely recalling from decayed memory
- context caching
The idea that our current languages might be as far as we get is absolutely demoralising. I don't want a tool to help me write pointless boilerplate in a bad language, I want a better language.
The companies valuing the expensive talent currently working on Google will be the winner.
Google and others are betting big right now, but I feel the winner might be those who watches how it unfolds first.
As a dev, I like it. It speeds up writing easy but tedious code. It's just a bit smarter version of the refactoring tools already common in IDEs...
Of course LLMs are a fantastic tool to improve productivity, but current LLM's cannot produce anything novel. They can only reproduce what they have seen.
And if we get 9 women we can produce a baby in a single month.
There's no guarantee such progression will continue. Indeed, there's much more evidence it is coming to a a halt.
More and more of the content generated since is LLM generated and useless as training data.
The models get worse, not better by being fed their own output, and right now they are out of training data.
This is why Reddit just went profitable, AI companies buy their text to train their models because it is at least somewhat human written.
Of course, even reddit is crawling with LLM generated text, so yes. It is coming to a halt.
It might be possible but will shareholders/investors foot the bill for the 80% that they still have to pay.
Why no youtube videos thought? Well, most dev you tubers are actual devs that cultivate an image of "I'm faster than LLM, I never re-read library references, I memorise them on first read" and do on. If they then show you a video how they forgot the syntax for this or that maven plugin config and how LLM fills it in 10s instead of a 5min Google search that makes them look less capable on their own. Why would they do that?
It is now very easy to sprinkle in regexes to validate user input , like email addresses, on every controller instead of using a central lib/utility for that.
In the hands of a skilled engineer it is a good tool. But for the rest it mainly serves to output more garbage at a higher rate.
Some people are touting this as a major feature. "I don't have to pull in some dependency for a minor function - I can just have AI write that simple function for me." I, personally, don't see this as a net positive.
The way it is now just leads to bloat and cruft.
Maybe I'm just old, but to me, LLMs feel like magic. A decade ago, anyone predicting their future capabilities would have been laughed at.
The discussion here seems to bare this out: CEO claims AI is magical, here the truth becomes that it’s just an auto-complete engine.
And so were a lot of markov chain based chatbots. Also Doretta, the microsoft AI/search engine chatbot.
Were they as good? No. Is this an iteration of those? Absolutely.
At Google, today, for sure.
I do believe we still are not across the road on this one.
> if this can be accompanied by an increase in software quality, which is possible. Right now its very hit and miss
So, is it really a smart move of Google to enforce this today, before quality have increased? Or did this set off their path to losing market shares because their software quality will deteriorate further over the next couple years?
From the outside it just seems Google and others have no choice, they must walk this path or lose market valuation.
That's what people are doing. The direction of travel over the most recent few (6-12) months is mostly flat.
The direction of travel when first introduced was a very steep line going from bottom-left to top-right.
We are not there anymore.
I'm excited about the possibilities and I still recoil at the refined marketer prose.
Will AI be able to detect bugs and back doors that require multiple pieces of code working together rather than being in a single piece of code? Humans have a hard time with this.
- Hypothetical Example: Authentication bugs in sshd that requires a flaw in systemd which then requires a flaw in udev or nss or PAM or some underlying library ... but looking at each individual library or daemon there are no bugs that a professional penetration testing organization such as the NCC group or Google's Project Zero would find. In other words, will AI soon be able to find more complex bugs in a year than Tavis has found in his career and will they start to compete with one another and start finding all the state sponsored complex bugs and then ultimately be able to create a map that suggests a common set of developers that may need to be notified? Will there be a table that logs where AI found things that professional human penetration testers could not?
Adversaries are already detecting issues tho, using proven means such as code review and fuzzing.
Google project zero consists of a team of rock star hackers. I don't see LLM even replacing junior devs right now.
Considering how terrible and frequently broken the code that the public facing Gemini produces, I'll have to be honest that that kind of scares me.
Gemini frequently fails at some fairly basic stuff, even in popular languages where it would have had a lot of source material to work from; where other public models (even free ones) sail through.
To give a fun, fairly recent example, here's a prime factorisation algorithm it produce for python:
# Find the prime factorization of n
prime_factors = []
while n > 1:
p = 2
while n % p == 0:
prime_factors.append(p)
n //= p
p += 1
prime_factors.append(n)
I bet I'd make more errors on my first try at it.
Perhaps even, there could be fewer Go programmers here and some just took a stab at it even though they don’t know the language. So it could just select for which forum has the most Go programmers. Hardly rigourous.
So I’d take that with a pinch of salt personally
If that doesn't satisfy, here's a similar one at leetcode.com: https://leetcode.com/problems/distinct-prime-factors-of-prod...
I would not expect a programmer of any seniority to churn stuff like that and have it working without testing.
I've been able to write one, not from memory but from first principles, any time in the last 40 years.
I'm happy you know math, but my point before this thread got derailed was that we're holding (coding) AI to a higher standard than actual humans, namely to expect to write bug-free code.
This seems like a very layman attitude and I would be surprised to find many devs adhering to this idea. Comments in this thread alone suggests that many devs on HN do not agree.
Testing for small prime factors is easy - brute force is your friend. Testing for large prime factors requires more effort. So the first trick is to figure out the bounds to the problem. Is it int32? Then brute-force it. Is it int64, where you might have a value like the Mersenne prime 2^61-1? Perhaps it's time to pull out a math reference. Is it longer, like an unbounded Python int? Definitely switch to something like the GNU Multiple Precision Arithmetic Library.
In this case, the maximum value is 1,000, which means we can enumerate all distinct prime values in that range, and test for its presence in each input value, one one-by-one:
# list from https://www.math.uchicago.edu/~luis/allprimes.html
_primes = [
2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59,
61, 67, 71, 73, 79, 83, 89, 97, 101, 103, 107, 109, 113, 127, 131,
137, 139, 149, 151, 157, 163, 167, 173, 179, 181, 191, 193, 197,
199, 211, 223, 227, 229, 233, 239, 241, 251, 257, 263, 269, 271,
277, 281, 283, 293, 307, 311, 313, 317, 331, 337, 347, 349, 353,
359, 367, 373, 379, 383, 389, 397, 401, 409, 419, 421, 431, 433,
439, 443, 449, 457, 461, 463, 467, 479, 487, 491, 499, 503, 509,
521, 523, 541, 547, 557, 563, 569, 571, 577, 587, 593, 599, 601,
607, 613, 617, 619, 631, 641, 643, 647, 653, 659, 661, 673, 677,
683, 691, 701, 709, 719, 727, 733, 739, 743, 751, 757, 761, 769,
773, 787, 797, 809, 811, 821, 823, 827, 829, 839, 853, 857, 859,
863, 877, 881, 883, 887, 907, 911, 919, 929, 937, 941, 947, 953,
967, 971, 977, 983, 991, 997]
def distinctPrimeFactors(nums: list[int]) -> int:
if __debug__:
# The problem definition gives these constraints
assert 1 <= len(nums) <= 10_000, "size out of range"
assert all(2 <= num <= 1000 for num in nums), "num out of range"
num_distinct = 0
for p in _primes:
for num in nums:
if num % p == 0:
num_distinct += 1
break
return num_distinct
import unittest
class TestExamples(unittest.TestCase):
def test_example_1(self):
self.assertEqual(distinctPrimeFactors([2,4,3,7,10,6]), 4)
def test_example_2(self):
self.assertEqual(distinctPrimeFactors([2,4,8,16]), 1)
def test_2_is_valid(self):
self.assertEqual(distinctPrimeFactors([2]), 1)
def test_1000_is_valid(self):
self.assertEqual(distinctPrimeFactors([1_000]), 2) # (2*5)**3
def test_10_000_values_is_valid(self):
values = _primes[:20] * (10_000 // 20)
assert len(values) == 10_000
self.assertEqual(distinctPrimeFactors(values), 20)
@unittest.skipUnless(__debug__, "can only test in debug mode")
class TestConstraints(unittest.TestCase):
def test_too_few(self):
with self.assertRaisesRegex(AssertionError, "size out of range"):
distinctPrimeFactors([])
def test_too_many(self):
with self.assertRaisesRegex(AssertionError, "size out of range"):
distinctPrimeFactors([2]*10_001)
def test_num_too_small(self):
with self.assertRaisesRegex(AssertionError, "num out of range"):
distinctPrimeFactors([1])
def test_num_too_large(self):
with self.assertRaisesRegex(AssertionError, "num out of range"):
distinctPrimeFactors([1_001])
if __name__ == "__main__":
unittest.main()
If I had no internet access, I would compute that table thusly:
_primes = [2]
for value in range(3, 1000):
if all(value % p > 0 for p in _primes):
_primes.append(value)
What kind of senior programmers do you work with who can't handle something like this?
EDIT: For fun I wrote an implementation based on sympy's integer factorization:
from sympy.ntheory import factorint
def distinctPrimeFactors(nums: list[int]) -> int:
distinct_factors = set()
for num in nums:
distinct_factors.update(factorint(num))
return len(distinct_factors)
def test_Mersenne(self):
self.assertEqual(distinctPrimeFactors(
[2**44497-1, 2,4,3,7,10,6]), 5)Assuming discussions don't happen in Slack, or Discord, or...
Did you know? How WEIRD.
How about you not harass other commenters with such arrogantly ignorant sarcastic questions?? Or is that part of corporate "for-profit" culture too????
So is marketing? So finance? So is petroleum engineering?
You were probably being rhetorical, but there are two problems:
- `p = 2` should be outside the loop
- `prime_factors.append(n)` appends `1` onto the end of the list for no reason
With those two changes I'm pretty sure it's correct.
def factorize(n):
...
return prime_factorsBut I don't understand this attempt to tell companies/persons that are successfully using AI that no they really aren't.
In my opinion, if they feel they're using AI successfully, the goal should be to learn from that.
I don't understand this need to tell individuals who say they are successfully using AI that, "no you aren't."
It feels like a form of denial.
Like someone saying, "I refuse to accept that this could work for you, no matter what you say."
The whole paradigm should change. If you are indeed responsible for developer tools, I would hope that you’re activity leveraging Claude 3.5 Sonnet and o1-preview.
It's always felt like having AI in the cloud for better autocomplete is a lot for a small gain.
Consider the fact that GPT-4 can generate valid XML (meaning balanced tags, quotes etc) in base64-encoded form. Without CoT, just direct output.
I don't know what model Copilot uses these days, but it constantly makes bracket mistakes in Python.
I’m pretty sure around 50% of the code I write is already auto-complete, without any AI.
We don’t need AI for this and it’s 10x the compute to do it slower with AI.
LLMs are useful but they aren’t a silver bullet. We don’t need to replace everything with it just because.
AI doesn't mean LLM after all. AI means 'a computer thing'.
On the other hand, that doesn't mean it doesn't count for the purpose of this press release/advertisement...
(because it's like 50k lines regenerated for every build, everywhere, all the time)
It's a different thing to what you're talking about, but it's one way I'd expect to see LLMs contribute a lot to productivity on larger codebases specifically.
When dealing with a fermenting pile of technical debt? I expect very little. LLM's don't have application-wide context yet.
AI is definitely revolutionizing our field, but the same people that said that no-code tools and all of the other hype-of-the-decade technologies would make developers jobless are actually the people AI is making jobless.
Generate an opinion piece about how AI is going to make developers jobless, using AI? Less than a minute. And you don't need to maintain that article, once it's published, it's done.
While there's a tsunami of AI-generated almost-there projects coming that need to be moved to a shippable and sellable state. So I'm more afraid about the kind of work I'm going to get while still getting paid handsomely for my skills, than ever being jobless as the only guy that really understands the whole stack from top to bottom.
One common misconception is that all LLMs are the same. The models are trained the same, but trained on wildly different datasets. Google, and more specifically the Google codebase is arguably one of the most curated, and iterated on datasets in existence. This is a massive lever for Google to train their internal code-gen models, that realistically could easily replace any entry-level or junior developer.
- Code review is another dimension of the process of maintaining a codebase that we can expect huge improvements with LLMs. The highly-curated commentary on existing code / flawed diff / corrected diff that Google possesses give them an opportunity to build a whole set of new internal tools / infra that's extremely tailored to their own coding standard / culture.
Nope
This is a massive, unsubstantiated leap.
In many cases it is counterproductive
I spent 12 years of my career in the Google codebase.
This assertion is technically correct in that google3 has been around for 20 years, and all code gets reviewed, but the implication that Google's codebase is a high-quality training set is not consistent with my experience.
The other 75% is the stuff you actually have to think about.
This feels like saying linters impact x0% of code. This just feels like an extension of that.
It's just a very advanced autocomplete, completely integrated into the internal codebase and IDE. You can read this on the research blog (maybe if everyone just read the blog).
e.g.
I start typing `var notificationManager`
It would suggest `= (Notification Manager) context.getSystemService(NOTIFICATION_MANAGER);`
If you've done Android then you know how much boilerplate there is to suggest.
I press Ctrl+Enter or something to accept the suggestion.
Voila, more than 50% of that code was written by AI.
> blindly committing AI code
Even before AI, no one blindly accepts autocomplete.
A lot of headline-readers seem to imagine some sort of semi-autonomous or prompt based code generation that writes whole blocks of code to then be blindly accepted by engineers.
I’m pretty sure the actual ratio is much lower than that. In other words, LLMs are currently not good enough to remove the majority of chores, even with the state of the art model trained on highly curated dataset.
Most of the code must be what could be snippets (opening files and handling errors with absl::, and moving data from proto to proto). One thing that doesn't help here, is that when writing for many engineers on different teams to read, spelling out simple code instead of depending on too many abstractions seems to be preferred by most teams.
I guess that LLMs do provide smarter snippets that I don't need to fill out in detail, and when it understands types and whether things compile it gets quite good and "smart" when it comes to write down boilerplate.
Everyone knows this was “made by AI” because there’s no way in hell I would ever have the time. These models might not be able to sit there and build an entire project from scratch yet, but if what you need is some help adding the next control panel page, Claude’s got your back on that.
If you’re not a developer chances are very high the code it produces will look passable but is actually worthless — or worse, it’s misleading and now a dev has to spend more time deciphering the task.
They should know because you told them so.
Having to decipher weird code only to discover it was not written by a human is not nice.
Doubtful. A decent fraction of the people reading it will guess that you've wasted your time writing incoherent nonsense in the jira. Engineers don't usually have much insight into what the C suite are doing. It would be a prudent move to spend the couple of seconds to write "something like this AI sketch:" before the copy&paste.
Fascinating point of view.
When someone who uses a product says it, there is a 50% chance of it being true, but when someone far away from the user says it, it is 100% promotion of product and setup for trust building for a future sale.
Not what I thought when I heard "AI coding", but seems pretty neat.
.... so what do you put in your scripts if not code?
I would imagine it always has.
>Google CEO says more than a quarter of the company's new code is created by AI
It may very well be starting to become apparent anyway :\
25% of coding is just the most basic boilerplate. I think of AI not as a thinking machine but as a 1000 WPM boilerplate typer.
If it is halucinating, you're trying to make it do stuff that is too complex.
Thats my main problem: for trivial things it works but isnt much better than conventional tools, for hard things it just produces incorrect code such that writing it from scratch barely makes a difference
What would it look like if I could have 3-500 snippets instead of 30. Those 300 are things that I do all over my codebase e.g. same basic where query but in the context of whatever function I am in, a click handler with the correct types for that purpose, etc.
There is no way I can have enough hotkeys or memorize that much, and I truly can't type faster than I can hit tab.
I don't need it to think for me. Most coding (front-end/back-end web) involves typing super basic stuff, not writing complex algorithms.
This is where the 10-20% speed-up comes in. On average I am just typing 20% faster by hitting tab.
Although, if we were to ignore all this for a second, you could also make similar estimates with, e.g., gzip: the higher the compression ratio attained, the more "verbose"/"fluffy" the code is.
Fun tangent: there are a lot of researchers who believe that compression and intelligence are equivalent or at least very tightly linked.
I'm not sure though. If it's copied a bunch of times, and it actually doesn't matter because each usecase of the copying is linearly independent, does it matter that it was copied?
Over time, you'd still see copies being changed by themselves show up as increased entropy
In fact it's the one thing they are explicitly designed to do, the rest is more or less a side-effect.
WinRAR can do that for you quite effectively.
I’m aware that the difference is that AI-generated code can be read and modified by humans. But that quantity is bad because humans have to understand it to read or modify it.
What’s the point of shorter code if you can’t trust it to do what it’s supposed to?
I’ll take 20 lines of code that do what they should consistently over 1 line that may or may not do the task depending on the direction of the wind.
fun story: today I had an LLM write me a non-trivial perl one-liner. It tried to be verbose but I insisted and it gave me one tight line.
I understand more code as being more edge cases
I can also suggest alternatives, like using existing library functions for things that might have been coded manually.
Whenever the problem requires thinking, it horribly fails because it cannot reason (yet). So unless this is also true for google devs, I cannot see that 25% number.
Besides, even google employees write a lot of boilerplate, especially android IIRC, not to mention simple but essential code, so AI can prevent carpal tunnel for the junior devs working on that.
... and forgot to count the Delete presses.
Is Google out there monitoring the IDE activity of every engineer, logging the amount of code created, by what, lines, characters, and how it was generated? Dubious.
A good chunk () of their code goes in a centralized repo, and is written via a centralized web IDE. So measuring everything you mentioned is not hard.
() Android, Chrome, and other similar projects are exceptions.
Yes, they'll miss AI-generated code that is copy pasted, so they only have a lower bound of AI-generated code.
For the example of generating an http app scaffolding from an openapi spec, it would probably account for at least 25% of the text in the generated source code. But I imagine this report would conveniently exclude the creation of the original source yaml driving the generator — I can’t imagine you’d save much typing (or mental overhead) trying to prompt a chatbot to design your api spec correctly before the codegen
Maybe my situation is unusual; I haven't written all that much code at Google lately, but what I do write is pretty tied to specific details of the program and the AI auto completion is just not that useful. Sometimes it auto completes a method signature correctly, but it never gets the body right (or even particularly close).
And it routinely making up methods or fields on objects I want to use is anti productive.
In that case those 25% are probably the very same 25% that were automatically generated by LTP based auto-completion.
1. Paste in my actual code.
2. Prompt: Write unit tests, test tables. Include scenarios: A, B, C, D, E. Include all other scenarios I left out, isolate suggestions for review.
I used to spend the majority of the coding time writing unit tests and mocking test data, now it's more like 10%.
> Prompt: Write unit tests
TDD in shambles. What you'd like is:
> Give your specs to some AI
> Get a test suite generated with all edge cases accounted for
> Code
I'll be looking into ways of running a local LLM for this purpose (code assistance in VS Code). I'm already really impressed with various quite large models running on my 32 GB Mac Studio M2 Max via Ollama. It feels like having a locally running chatgpt.
The Apple Silicon Macs have this shared memory between CPU and GPU that let's the (relatively underpowered GPU, compared to a decent Nvidia GPU) run these models at decent speeds, compared with a CPU, when using llama.cpp.
This should all get dramatically better/faster/cheaper within a few years, I suspect. Capitalism will figure this one out.
I assume, then, that the primary goal would be to drop in the beefiest GPU possible when on windows/linux?
I'm sure that there are other much more knowledgeable people here though, on this topic.
In Windows and Linux, yes you'll want at least 12GB of VRAM to have much of any utility but the beefiest consumer GPUs are still topping out at 24GB which is still pretty limiting.
It immediately works out of the box and that's it. I've been using local LLMs on my laptop for a while, it's pretty nice.
The only thing you really need to worry about is VRAM. Make sure your GPU has enough memory to run your model and that's pretty much it.
Also "open webui" is the worst project name I've ever seen.
The AI is fine, but every time it makes a little mistake that I have to correct it really breaks my flow. I might type a lot more boilerplate without it but I get better flow and overall that saves me time with less mistakes.
But you make an interesting point: eventually AI will be making for other AI's + machines, and human verification can be an after thought.
You tell investors that AI is freaking magic and going to usher in an age of savings and productivity gains.
You tell your developers that it's a neat autocomplete, they should use carefully.
Some say „this is mere tab completion“, some say „it won’t replace the senior engineer.“
I can remember how many fiercely argued 2 years ago that GenAI and Copilot are producing garbage. But here we are: These systems improve the workflow of creating / editing code enormously. You seniors might not be affected, but there are endless many scenarios where it replaces the junior who‘d write code to transform data, write scripts, write one-off scripts, or even write boilerplate, test code and what not.
And this is only after a short time. I cannot even imagine what we‘ll have ten years from now where we can propably have much larger context windows where the system can „unterstand“ the whole code base, not just parts.
I am sorry for low level engineering jobs, but I am super exited as well.
With GebAI I have been writing super complex Elisp code to automate workflow in Emacs, or VBA scripts in Excel, or Bash scripts I wouldn’t have otherwise been able to write, or JavaScript, or quickly write Python code to solve very tricky problems (and I am very high level in Python), or even React code for web apps for my personal use.
The future looks exiting to me.
This is the disconnect. I, along with others, haven't seen this yet. I'm begging to see it because I'd love to automate my work away, but I can't. This comment comes off as hand-wavy to me because it says "here we are" as if Google saying their AI works is evidence itself and not a statement that requires evidence.
Google, and really, the whole financial machine has a vested interest playing up the potential of AI. Unfortunate that it isn’t being given time to grow organically.
Financial incentives at large companies are not aligned with low volumes of code. There are no rewards for less code. People get rewarded for another bullshit framework to slap on their resume. Box me in, no, cube me in to a morass of a thick ingress layer, that uses 1/8th of my CPU.
Claude 3.5-sonnet (latest) is barely able to stay coherent on 500 LOC files, and easily gets tripped up when there are several files in the same directory.
I have tried similarly with o1-preview and 4o, and gemini pro...
If google is using a 5M token context window LLM with 100k+ token-output trained on all the code that is not public... then I can believe this claim.
This just goes to show how critical of an issue this is that these models are behind closed doors.
How is competitive advantage, using in-house developed/funded tools, a critical issue? Every company has tools that only they have, that they pay significantly for to develop, and use extensively. It's can often be the primary thing that really differentiates companies who are all doing similar things.
His position seems secure despite these missteps, which highlights an interesting double standard: there appears to be far more tolerance for strategic failures at the CEO level compared to the rigorous performance standards expected of engineering staff.
NO! False. I can confirm they are not. I've known of several major obvious unfixed bugs/flaws in Google apps for years. and in the last year or so especially theres been an explosion in the number of head-scratching, jaw-dropping fails and UX anti-patterns in their code. GMail, Search, Maps and Android are now riddled with them.
on Sundar Pichai's watch he's been devolving Google to be yet another Microsoft type in terms of quality, care and taste.
My conclusion is that we are at the first wave of a split between those who use LLMs to augment their abilities and knowledge, and those who delay. In cyberpunk terminally, it's aug-tech, not real AGI. (and the lesser ones code abilities and simpler the task, the more benefit, it's an accelerator)
What missing is that code being written by AI may have less of an impact than dataset that are developed or refined by AI. Consider examples like a utility function's coefficients, or the weights of a model.
As these are aggressively tuned using ML feedback, they'll influence far more systems than raw code.
1. Generate unit tests for modules which are already written to be tested 2. Generate documentation for interfaces
Both of these require quite deep knowledge in what to write, then it simply documents and fills in the blanks using the context which already has been laid out.
I will be curious to see if it has any impact positive or negative over a couple of years.
Will the code be more secure since the AI does not make the mistakes humans do?
Or will the code, not well enough understood by the employees, exposes exploits that would not be there?
Will it change average up time?
More lines == more shit to maintain. Complex lines == the shit is unmanageable.
But wall street investors love simplistic narratives such as More X == More revenue. So here we are. Pretty clever marketing imo.
These statements are brilliant.
Search Copy-Paste as a Service is hiding a deeper issue.
Alphabet ($GOOG) 2024 Q3 earnings release
How do they know this? At face value, it sounds like alot, but it only says "new code generated". Nothing about code making it into source control or production, or even which parts of googles vast business units.
For all we know, this could be the result of some internal poll "Tell us if you've been using Goose recently" or some marketing analytics on the Goose "Generate" button.
It's puff piece to put Google back in the lime light, and everyone is lapping it up.
:)
If this stuff really works as well as these companies claim it does, wouldn’t their entire workforce excitedly be using these tools already?
I expect these tools to improve productivity for new-ish developers, however for anyone that is literate in a programming language the effect is marginal at best ("Copilot pause" etc.)
You have to establish that the CEO is actually aware of the reality and is interested in accurately conveying that to you. As far as I can tell there is absolutely no reason to believe any part of this.
Not to mention Elon publicly demonstrated losing 80% of staff when he took over twitter and - you can complain about his management all you want - as someone who's been using it the whole way through, from a technical POV their downtimes and software quality has not been any worse and they're shipping features faster. A lot of software companies are overstaffed, especially Google who has spent years paying people to make projects just to get a PO promoted, then letting the projects rot and die to be replaced by something else. That's a lot of useless work being done.
which puts it in line with previous code-generation technologies i would imagine. I wonder which of these increased productivity the most?
- Assembly Language
- early Compilers
- databases
- graphics frameworks
- ui frameworks (windows)
- web apps
- code generators (rails scaffolding)
- genAI
The gap between "high level assembly" and "compiled language" is about as large as it gets.
Is that a good thing though? We should work and making code small and easy to manage without AI tools.
Seriously. The penchant for outright ignoring user search terms, relentlessly forcing irrelevant or just plain wrong information on users, and the obnoxious UI changes on YouTube! If I'm watching a video on full screen I have explicitly made it clear that I want YouTube to only show me video! STOP BRINGING UP THE FUCKING VIDEO DESCRIPTION TO TAKE UP HALF THE SCREEN IF I TRY TO BRIEFLY SWIPE TO VIEW THE TIME OR READ A MESSAGE.
I have such deep-seated contempt for AI and it's products for just how much worse it makes people's lives.
Then again, their devices are also coming out with known fatal design flaws, like not being able to make phone calls, or the screen going black permanently.
Like, isn't this announcement a terrible indictment of how inexperienced their engineers are, or how trivial the problems they solve are, or both?
Even if their engineers were inexperienced that wouldn't be an indictment in itself so long as they had a sufficient necessary amount of shallow work. Using all experienced engineers to do shallow work is just inefficient, like having brain surgeons removing bunions. Automation is basically a way to transform deep work to a producer of "free" shallow work.
That said, the real impressive thing with code isn't in its creation but in its ability to losslessly delete code and maintain or improve functionality.
Now replace all this and much more with 'AI'. If they said AI helped them increase say ad effectivity by 3-5%, I'll start paying attention.
Of obviously flawed corporate structures. This CEO has no particular programming expertise and most of his companies profits do not seem to flow from this activity. I strongly doubt he has a grip on the actual facts here and is uncritically repeating what was told to him in a meeting.
He should, given his position, been the very _first_ person to ask the questions you've posed here.
This bothers me. I completely understand the conversational aspect - "what approach might work for this?", "how could we reduce the crud in this function?" - it worked a lot for me last year when I tried learning C.
But the vast majority of AI use that I see is...not that. It's just glorified, very expensive search. We are willing to burn far, far more fuel than necessary because we've decided we can't be bothered with traditional search.
A lot of enterprise software is poorly cobbled together using stackoverflow gathered code as it is. It's part of the reason why MS Teams makes your laptop run so hot. We've decided that power-inefficient software is the best approach. Now we want to amplify that effect by burning more fuel to get the same answers, but from an LLM.
It's frustrating. It should be snowing where I am now, but it's not. Because we want to frivolously chase false convenience and burn gallons and gallons of fuel to do it. LLM usage is a part of that.
Traditional search (at least on the web) is dying. The entire edifice is drowning under a rapidly rising tide of spam and scam sites. No one, including Google, knows what to do about it so we're punting on the whole project and hoping AI will swoop in like deus ex machina and save the day.
The problem with Google search is that it indexes all the web, and there's (as you say) a rising tide of scam and spam sites.
The problem with AI is that it scoops up all the web as training data, and there's a rising tide of scam and spam sites.
You make this claim with such confidence, but what is it based on?
There have always been hordes of spam and scam websites. Can you point to anything that actually indicates that the ratio is now getting worse?
No, there haven't always been hordes of spam and scam websites. I remember the web of the 90s. When Google first arrived on the scene every site on the results page was a real site, not a spam/scam site.
Google results are polluted with spam because it is more profitable for Google. This is a conscious decision they made five years ago.
Then why are DuckDuckGo results also (arguably even more so) polluted with spam/scam sites? I doubt DDG is making any profit from those sites since Google essentially owns the display ad business.
Google not only has multiple monopolies, but a cut and dry perverse incentive to produce lower quality results to make the whole session longer instead of short and effective.
Google wasn’t crushed by spam, they decided to stop doing text search and build search bubbles that are user specific, location-specific, decided to surface pages that mention search terms in metadata instead of in text users might read, etc. Oh yeah, and about a decade before LLMs were actually usable, they started to sabotage simple substring searches and kind of force this more conversational interface. That’s when simple search terms stopped working very well, and you had to instead ask yourself “hmm how would a very old person or a small child phrase this question for a magic oracle”
This is how we get stuff like: Did you mean “when did Shakespeare die near my location”? If anyone at google cared more about quality than printing money, that thirsty gambit would at least be at the bottom of the page instead of the top.
Google’s verbatim search option roughly does that for me (plus an ad blocker that removes ads from the results page). I have it activated by default as a search shortcut.
(To activate it, one can add “tbs=li:1” as a query parameter to the Google search URL.)
Thank you, this is almost life-alteringly good to know.
Seems that Firefox on mobile allows editing the url for most pages, but on google search results pages, the url bar magically turns into a did-you-mean alternate search selector where I cannot see nor edit a url. Surprised but not surprised.
Sure, there’s a work around for this too, somehow. But I don’t want to spend my life collecting and constantly updating a huge list of temporary hacks to fix things that others have intentionally broken.
Yes, it’s annoying that you can’t set it as the default on Google search itself.
By the time I graduated highschool you already couldn't trust that Boolean operators would be treated literally. By the time I graduated college, they basically didn't seem to do anything, at best a weak suggestion.
Nowadays quotes don't even seem to be consistently honored.
I suspect the real problem is that search engines ceased being search engines when they stopped taking things literally and started trying to interpret what people mean. Then they became some sort of poor man's AI. Now that we have LLMs, of course it is going to replace the poor excuse for search engines that exist today. We were heading down that road already, and it actually summarizes what is out there.
Then Google decided to start enforcing that, because they had this idea that they would be able to divine your "intent" from a "natural question" rather than just matching documents including your search terms.
Even more naive, but my personal preference: just ban all advertising. The fact that people will pay for ChatGPT implies people will also pay for good search if the free alternative goes away.
For example: a beginner developer is possibly better served by some SEO-heavy tutorial blog post; an experienced developer would prefer results weighted towards the official docs, the project’s bug tracker and mailing list, etc. But since less technical and non-technical people vastly outnumber highly technical people, Google and Bing end up focusing on the needs of the former, at the cost of making search worse for the later.
One positive about AI: if an AI is doing the search, it likely wants the more advanced material not the more beginner-focused one. It can take more advanced material and simplify it for the benefit of less experienced users. It is (I suspect) less likely to make mistakes if you ask it to simplify the more advanced material than if you just gave it more beginner-oriented material instead. So if AI starts to replace humans as the main clients of search, that may reverse some of the pressure to “dumb it down”.
I mostly agree with your interesting comment, and I think your analysis basically jives with my sibling comment.
But one thing I take issue with is the idea that this type of thing is a good faith effort, because it’s more like a convenient excuse. Explaining substring search or even include/exclude ops to children and grandparents is actually easy. Setting preferences for tutorials vs API docs would also be easy. But companies don’t really want user-directed behavior as much as they want to herd users to preferred content with algorithms, then convince the user it was their idea or at least the result of relatively static ranking processes.
The push towards more fuzzy semantic search and “related content” everywhere is not to cater to novice users but to blur the line between paid advertisement and organic user-directed discovery.
No need to give megacorp the benefit of the doubt on stuff like this, or make the underlying problems seem harder than they are. All platforms land in this place by convergent evolution wherein the driving forces are money and influence, not insurmountable technical difficulties or good intentions for usability.
Good luck finding those, you end op with SEO spam and clone page spam. These days you have to look for unobvious hidden meanings which only relate to your exact problem to find what you are looking for.
I have the strong feeling search these days is back to the Altavista era. You'd have to use trickery to find what you were looking for back then as well. Too bad + no longer works in google due to their stupid naming of a dead product (no, literal is not the same and no replacement).
That's not my experience at all. While there are scammy sites, using the search engines as an index instead of an oracle still yields useful results. It only requires to learn the keywords which you can do by reading the relevant materials .
I'm sure they can. But they have no incentive. Try to Google an item, and it will show you a perfect match of sponsored ads and some other not-so-relevant non-sponsored results
Tailoring/retraining the main search AI will be so much more expensive that retraining the spam special purpose AIs.
Since the collapse of Internet search (rose tinted hindsight - was it ever any good?) I have been using a LLM as my syntax advisor. I pay for my own tokens, and I can say it is astonishingly cheap
It is also very good.
In other words, it's a skill issue. LLMs can only make this worse. Hiring unskilled programmers and giving them a machine for generating garbage isn't the way. Instead, train them, and reject low quality work.
In my opinion the reason we get enterprise spaghetti is largely due to requirement issues and scope creep. It's nearly impossible to create a streamlined system without knowing what it should look like. And once the system gets to a certain size, it's impossible to get business buy-in to rearchitect or refactor to the degree that is necessary. Plus the full requirements are usually poorly documented and long forgotten by that time.
> it's impossible to get business buy-in to rearchitect or refactor to the degree that is necessary
That's a choice. There are some other options:
- Simply don't get business buy-in. Do without. Form a terrorist cell within your organization. You'll likely outpace them. Or you'll get fired, which means you'll get severance, unemployment, a vacation, and the opportunity to apply to a job at a better company.
- Fight viciously for engineering independence. You business people can do the businessing, but us engineers are going to do the engineering. We'll tell you how we'll do it, not the other way.
- Build companies around a culture of doing good, consistent work instead of taking expedient shortcuts. They're rare, but they exist!
Or simply find a position in an industry or department where you commonly have more independence. In my opinion this fight is not worth it - look for another position instead is typically easier.
Congratulations, you just refactored out a use case which was documented in a knowledge base which has been replaced by 3 newer ones since then, happens once every 18 months and makes the company go bankrupt if it isn't carried out promptly.
The type of junior devs who think that making code tidy is fixing the application are the type of dev who you don't let near the heart of the code base, and incidentally the type who are best replaced with code gen AI.
And regardless, the way you prevent loss of important functionality isn't by hoping people read docs that no longer exist. It's by writing coarse-grained tests that makes sure the software does the important things. If a programmer wants to change something that breaks a test like that, they go ask a product manager (or whatever you call yours) if that feature still matters.
And if nobody can say whether a feature still matters, the organization doesn't have a software problem, it has a serious management problem. Not all the coding techniques in the world can fix that.
> ...
> train them, and reject low quality work.
I agree very strongly with both of these points.
But I've observed a truth about each of them over the last decade-plus of building software.
1) very few people approach the field of software engineering with anything remotely resembling rigor, and
2) there is often little incentive to train juniors and reject subpar output (move fast and break things, etc.)
I don't know where this takes us as an industry? But I feel your comment on a deep level.
I’ve had the good fortune of getting to write some firmware that will likely work well for a long time to come, but I find most things being written on computers are written with (or very close to) the minimum care possible in order to get the product out. Clean up is intended but rarely occurs.
I think we’d see real benefits from doing a better job, but like many things, we fail to invest early and crave immediate gratification.
I have this one opinion which I would not say at work:
In software development it's easy to feel smart because what you made "works" and you can show "effects".
- Does it wrap every failable condition in `except Exception`? Uhh, but look, it works.
- Does it define a class hierarchy for what should be a dictionary lookup? It works great tho!
- Does it create a cyclic graph of objects calling each other's methods to create more objects holding references to the objects that created them? And for what, to produce a flat dictionary of data at end of the day? But see, it works.
this is getting boring, maybe just skip past the list
- Does it stuff what should be local variables and parameters in self, creating a big stateful blob of an object where every attribute is optional and methods need to be called in the right order, otherwise you get an exception? Yes, but it works.
- Does it embed a browser engine? But it works!
The programmer, positively affirmed, continues spewing out crap, while the senior keep fighting fires to keep things running, while insulating the programmer from the taste of their own medicine.
But more generally, it's hard to expect people to learn how to solve problems simply if they're given gigantic OO languages with all the features and no apparent cost to any of them. People learn how to write classes and then never learn get good at writing code with a clear data flow.
Even very bright people can get fall for this trap because engineering isn't just about being smart but about using intelligence and experience to solve a problem while minmaxing correctly chosen properties. Those properties should generally be: dev time, complexity (state/flow), correctness, test coverage, ease of change, performance (anything else?). Anyway, "Affirming one's opinions about how things should be done" isn't one of them.
This is a huge problem. I don't know where it comes from, I think maybe sort of learned helplessness? Like, if systems are so complex that you don't believe a single person can understand it then why bother trying anyway? I think it's possible to inspire people to not accept not understanding. That motivation to figure out what's actually happening and how things actually work is the carrot. The stick is thorough, critical (but kind and fair) code--and, crucially, design--review, and demanding things be re-done when they're not up to par. I've been extremely lucky in my career to have had senior engineers apply both of these tools excellently in my general direction.
> 2) there is often little incentive to train juniors and reject subpar output (move fast and break things, etc.)
One problem is our current (well, for years now) corporate culture is this kind of gig-adjacent-economy where you're only expected to stick around for a few years at most and therefore in order to be worth your comp package you need to be productive on your first day. Companies even advertise this as a good thing "you'll push code to prod on your first day!" It reminds me of those scammy books from when I was a kid in the late 90s "Learn C In 10 Days!".
I think it's a bunch of things, but one legitimate issue is that software is stupidly complex these days. I had the advantage of starting when computers were pretty simple and have had a chance to grow along with it. (And my dad started when you could still lift up the hood and look at each bit. [1])
When I'm working with junior engineers I have a hard time even summing up how many layers lie beneath what they're working on. And so much of what they have to know is historically contingent. Just the other day I had to explain what LF and CR mean and how it relates to physical machinery that they probably won't see outside of a museum: https://sfba.social/@williampietri/113387049693365012
So I get how junior engineers struggle to develop a belief that the can sort it all out. Especially when so many people end up working on garbage code, where little sense is to be had. It's no wonder so many turn to cargo culting and other superstitious rituals.
Ahh, well, in order to save money, training is done via an online class with multiple choice questions, or, if your company is like mine and really committed to making sure that you know they take your training seriously, they put portions of a generic book on 'tech Z' in pdf spread spread over a drm ridden web pages.
As for code, that is reviewed, commented and rejected by llms as well. It is used to be turtles. Now it truly is llms all the way down.
That said, in a sane world, this is what should be happening for a company that actually wants to get good results over time .
I don't think finding such programmers is really difficult. What is difficult is finding such people if you expect them to be docile to incompetent managers and other incompetent people involved in the project who, for example, got their position not by merit and competence, but by playing political games.
The way I explain this to managers is that software development is unlike most work. If I'm making widgets and I fuck up, that widget goes out the door never to be seen again. But in software, today's outputs are tomorrow's raw materials. You can trade quality for speed in the very short term at the cost of future productivity, so you're really trading speed for speed.
I should add, though, that one can do the rigorous thinking before or after the doing, and ideally one should do both. That was the key insight behind Martin Fowler's "Refactoring: Improving the Design of Existing Code". Think up front if you can, but the best designs are based on the most information, and there's a lot of information that is not available until later in a project. So you'll want to think as information comes in and adjust designs as you go.
That's something an LLM absolutely can't do, because it doesn't have access to that flow of information and it can't think about where the system should be going.
This is an important point. I don't remember where I read it, but someone said something similar about taking a loss on your first few customers as an early stage startup--basically, the idea is you're buying information about how well or poorly your product meets a need.
Where it goes wrong is if you choose not to act on that information.
Without redoing their work or finding a way to have deep trust (which is possible, but uncommon at a bigcorp) it's hard enough to tell who is earnest and who is faking it (or buying their own baloney) when it comes to propositions like "investing in this piece of tech debt will pay off big time"
As a result, if managers tend to believe such plans, bad ideas drive out good and you end up investing in a tech debt proposal that just wastes time. Burned managers therefore cope by undervaluing any such proposals and preferring the crappy car that at least you know is crappy over the car that allegedly has a brand new 0 mile motor on it but you have no way of distinguishing from a car with a rolled back odometer. They take the locally optimal path because it's the best they can do.
It's taken me 15 years of working in the field and thinking about this to figure it out.
The only way out is an organization where everyone is trusted and competent and is worthy of trust, which again, hard to do at most random bigcorps.
This is my current theory anyway. It's sad, but I think it kind of makes sense.
Being good at the NATO style of management means focusing on the big picture--what, when, why--and leaving how to the people actually doing it.
There is no incentive to do it. I worked that way, focused on quality and testing and none of my changes blew up in production. My manager opined that this approach is too slow and that it was ok to have minor breakages as long as they are fixed soon. When things break though, it's blame game all around. Loads of hypocrisy.
That's because traditional search fucking sucks balls.
He couldn't pronounce the name of the extension, apparently not noticing that trgm == trigram, or what that might even be. Copying the output from the LLM and pasting it into a PR didn't result in anything other than him checking off a box, moving a ticket in Jira, and then onto the next thing--not even a pretense of being curious about what any of it all meant. But look at those query times now!
It's been possible for a while to shut off your brain as a programmer and blindly copy-paste from StackOverflow etc., but the level of enablement that LLMs afford is staggering.
"How do I do a for loop" though is a waste of time and energy and should be put into a search engine. There is no need to use the inefficient power needs of an LLM to answer that question. The search engine will have cached the results of that question, leading to a much faster discovery of the answer, and less power draw to do it, whereas an LLM needs to ponder your question EVERY. SINGLE. TIME. A huge waste.
Stop using LLMs for simple things.
Traditional search was only Google, and Google figured out that they don't need to improve their tools to make it better, because everyone will continue to use it as a force of habit (google is a verb!). Traditional search is being abandoned because traditional search isn't good enough for the kinds of search we need (also, while google may claim their search is very useful, people rarely search stuff nowadays, instead prefer being passively fed content via recommendations algorithm (that also use AI!))
In fact, I'd say I use AI more for documentation than I do for code itself, because AI generated documentation is often superior to official documentation.
In the end, these things shouldn't be necessary (or barely necessary) if we had well constructed languages, frameworks, libraries and documentation, but it appears like it's easier to build AI than to make things non-convoluted in the first place.
From what I've seen on Google Cloud, both as a user and from leaked source code, 25% of their code is probably just packing and unpacking of protobufs.
God I miss that, thanks for the other person on HN introducing me to projen. Yeoman wasnt cutting it.
These days I write a surprising amount of shell script and awk with LLMs. I review and adapt it, of course, but for short snippets of low context scripting it's been a huge time saver. I'm talking like 3-4, up to 20 lines of POSIX shell.
Idk. Some day I'll actually learn AWK, and while I've gotten decent with POSIX shell (and bash), it's definitely been more monkey see monkey do than me going over all the libraries and reference docs like I did for python and the cpp FAQ.
The way google works, the person changing an interface is responsible for updating all dependent code. They create PRs which are then sent to code owners for approval. For lower-level dependencies, this can involve creating thousands of PRs across hundreds of projects.
Google has had tooling to help with these large-scale refactors for decades, generally taking the form of static analysis tools. However, these would be inherently limited in their capability. Manual PR authoring would still be required in many cases.
With this background, LLM code gen seems like a natural tool to augment Google's existing process.
I expect Google is currently executing a wave of newly-unblocked refactoring projects.
If anyone works/worked at google, feel free to correct me on this.
If we’re guessing what code is easiest and largest proportion of codebase to write, my first guess would be test suites. Lots of lines of repetitive code patterns that repeat and AI is decent at dealing with
Well, just in the last 24 hours, ChatGPT gave me solutions to some relatively complex problems that turned out to be significantly wrong.
Did that mean it was a complete waste of my time? I’m not sure. Its broken code gave me a starting point for tinkering and exploring and trying to understand why it wasn’t working (even if superficially it looked like it should). I’m not convinced I lost anything by trying its suggestions. And I learned some things in the process (e.g. asyncio doesn’t play well together with Flask-Sock)
We all have probably 25% or more trivial code. AI is great for that. I have X (table structure, model, data, etc) and I want to make Y with it. A lot of code is pretty much mindless shuffling data around.
The other thing is good for is anything pretty standard. If I'm using a new technology and I just want to get started with whatever is the best practice, it's going to do that.
If I ever have to do PowerShell (I hate PowerShell), I can get AI to generate pretty much whatever I want and then I'm smart enough to fix any issues. But I really don't like starting from nothing in a tech I hate.
But for me it massively improved all the boilerplate generic work. A lot of those things which are just annoying work, but not interesting.
Then I can focus on the bigger things, on the important parts.
I’m pretty sure they weren’t the first and there’ve been others we didn’t know about. So now I don’t ask lead-in questions anymore. Surprisingly, it doesn’t seem to make much of a difference and I don’t need to get burned again.
How do you quantify "new code" - is it by lines of code or number of PRs/changesets generated? I can easily see it being the latter - if an AI workflow suggests 1 naming-change/cleanup commit to your PR made of 3 other human-authored commits, has it authored 25% of code? Arguably, yes - but it's trivial code that ought to be reviewed by humans. Dependabot is responsible for a good chunk of PRs already.
Having a monorepo brings plenty of opportunities for automation when refactoring - whether its AI, AST manipulation or even good old grep. The trick is not to merge the code directly, but have humans in the loop to approve, or take-over and correct the code first.
Honestly, I think it will become a better Intellisense but not much more. I'm a little excited because there's going to be so many people buying into this, generating so much bad code/bad architecture/etc. that will inevitably need someone to fix after the hype dies down and the rug is pulled, that I think there will continue to be employment opportunities.
We also have the ceremonial layers of certain forms of corporate architecture, where nothing actually happens, but the steps must exist to match the holy box, box cylinder architecture. Ceremonial input massaging here, ceremonial data transformation over there, duplicated error checking... if it's easy for the LLM to do, maybe we shouldn't be doing it everywhere in the first place.
I don't know that I've ever even met a developer who wants to be writing endless pools of trivial boilerplate instead of meaningful code. Even the people at work who are willing to say they don't want to deal with the ambiguity and high level design stuff and just want to be told what to do pretty clearly don't want endless drudgery.
We want to control code at the call site, boilerplate helps with that by being locally modifiable.
We also want to systematize chunks of code so that they don’t flicker around and mess with a reader.
We wanted this since forever and no one does anything because anything above simple text completion is traditionally seen as an overkill, not the true way, not unix, etc. All sorts of stubborn arguments.
This can be solved by simply allowing code trees instead of lines of code (tree vs table). You drop a boilerplate into code marked as “boilerplate ‘foo’ {…}” and edit it as you see fit, which creates a boilerplate-local patch. Then you can instantly see diffs, find, update boilerplates, convert them to and from regular functions, merge best practices from boilerplate libraries, etc. Problem solved.
It feels like the development itself got collectively stuck in some stupid principles that no one dares to question. Everything that we invent stumbles upon the simple fact that we don’t have any sensible devtime structure, apart from this “file” and “import file” dullness.
When I hear that most code is trivial, I think of this as a language design or a framework related issue making things harder than they should be.
Throwing AI or generates at the problem just to claim that they fixed it is just frustrating.
This was one of my thoughts too. If the pain of using bad frameworks and clunky languages can be mitigated by AI, it seems like the popular but ugly/verbose languages will win out since there's almost no point to better designed languages/framework. I would rather a good language/framework/etc where it is just as easy to just write the code directly. Similar time in implementation to a LLM prompt, but more deterministic.
If people don't feel the pain of AI slop why move to greener pastures? It almost encourages things to not improve at the code level.
Just as an example, I have "service" functions. They're incredibly simple, a higher order function where I can inject the DB handler, user permissions, config, etc. Every time I write one of these I have to import the ServiceDependencies type and declare which dependencies I need to write the service. I now spend close to zero time doing that and all my time focusing on the service logic. I don't see a downside to this.
Most of my business logic is done in raw SQL, which can be complex, but the autocomplete often helps there too. It's not helping me figure out the logic, it's simply cutting down on my typing. I don't know how anyone could be offered "do you want to have type significantly less characters on your keyboard to get the same thing done?" and say "no thanks". The AI is almost NEVER coding for me, it's just typing for me and it's awesome.
I don't care how lean your system is, there will at least be repetition in how you declare things. There will be imports, there will be dependencies. You can remove 90% of this repetitive work for almost no cost...
I've tried to use ChatGPT to "code for me", and I agree with you that it's not a good option if you're trying to do anything remotely complex and want to avoid bugs. I never do this. But integrated code suggestions (with Supermaven, NOT CoPilot) are incredibly beneficial and maybe you should just try it instead of trying to come up with theoretical arguments. I was also a non-believer once.
Regardless, I do wonder how accurate those successful reports are. Do people take LLM output, use it verbatim, not notice subtle bugs, and report that as success?
Unless you're the type of programmer that is writing sabots all day (connecting round pegs into square holes between two data sources) you've got to be very critical of what these things are spitting out.
If you just let it generate and run the code... yeah, probably, since you won't catch the issues at compile time.
This, imho, is what is happening. In the olden days, when StackOverflow + Google used to magically find the exact problem from the exact domain you needed every time - even then you'd often need to sift through the answers (top voted one was increasingly not what you needed) to find what you needed, then modify it further to precisely fit whatever you were doing. This worked fine for me for a long time until search rendered itself worthless and the overall answer quality of StackOverflow has gone down (imo). So, we are here, essentially doing the exact same thing in a much more expensive way, as you said.
Regarding future employment opportunities - this rot is already happening and hires are coming from it, at least from what I'm seeing in my own domain.
I've been using LLMs for about a month now. It's a nice productivity gain. You do have to read generated code and understand it. Another useful strategy is pasting a buggy function and ask for revisions.
I think most programmers who claim that LLMs aren't useful are reacting emotionally. They don't want LLMs to be useful because, in their eyes, that would lower the status of programming. This is a silly insecurity: ultimately programmers are useful because they can think formally better than most people. For the forseeable future, there's going to be massive demand for that, and people who can do it will be high status.
That's a bold statement, and incorrect, in my opinion.
At a junior level software development can be about churning out trivial code in a previously defined box. I don't think its fair to call that 'most programming'.
Most of the time, when I am typing code, the code I am producing is trivial, however.
I am just pointing out that the thread parent started his logical climb at a step one that is incorrect: 'Most programming is trivial'.
Given that they got it wrong on step one, how good do you thing step ten is?
I think that’s exactly right. I used to have to create the puzzle pieces and then fit them together. Now, a lot of the time something else makes the piece and I’m just doing the fitting together part. Whether there will come a day when we just need to describe the completed puzzle remains to be seen.
Even with the debugging example, if I just read what I wrote I'll find the bug because I understand the language. For more complex bugs, I'd have to feed the LLM a large fraction of my codebase and at that point we're exceeding the level of understanding these things can have.
I would be pretty happy to see an AI that can do effective code reviews, but until that point I probably won't bother.
I don't think that's true. Most programmers I speak to have been keen to try it out and reap some benefits.
The almost universal experience has been that it works for trivial problems, starts injecting mistakes for harder problems and goes completely off the rails for anything really difficult.
I’ve been seeing the complete opposite. So it’s out there.
As an example of not being production ready: I recently tried to use ChatGPT-4 to provide me with a script to manage my gmail labels. The APIs for these are all online, I didn't want to read them. ChatGPT-4 gave me a workable PoC that was extremely slow because it was using inefficient APIs. It then lied to me about better APIs existing and I realized that when reading the docs. The "vibes" outcome of this is that it can produce working slop code. For the curious I discuss this in more specific detail at: https://er4hn.info/blog/2024.10.26-gmail-labels/#using-ai-to...
Does a carpenter blame their hammer when it fails to drive in a screw?
However this laser measuring tool is accurate within a range. There's a lot of factors that affect it's accuracy like time of day, how you hold it, the material you point it at, etc. Sometimes these accuracy errors are minimal, sometimes they are pretty big. You end up getting a lot of measurements that seem "close enough". but you still need to ask if each one is correct. "Measure Twice, Cut Once" begins to require one measurement with the laser tool and once with the conventional tool when accuracy matters.
One could have a convoluted analogy where the carpenter has an electric hammer that for some reason has a rounded head that does cause some number of nails to not go in cleanly, but I like my analogy better :)
That's the exact problem. I have plenty of screwdrivers but there's so much pressure from people not in carpentry telling me to use this shiny new army Swiss knife contraption. Will it work? Probably, if I'm just screwing in a few screws. Would I readily abandon my set of precision built, magnetic tip, etc. Screwdriver set for it? Definitely not.
I'm sure it's great for non-carpenters to have so many tools in so small a space. But I developed skills and tools already. My job isn't just to screw in a few screws a day and call it quits. People wanting to replace me for a quarter the cost for this Swiss army carpenter will quickly see a quality difference and realize why it's not a solution to everything.
Or in the software sense, maybe they are fine with unlevel shelves and hanging nails in carpet. It's certainly not work I'd find acceptable.
LLMs are great at translating already-rigorously-thought-out pseudocode requirements, into a specific (non-esoteric) programming language, with calls to (popular) libraries/APIs of that language. They might make little mistakes — but so can human developers. If you're good at catching little mistakes, then this can still be faster!
For a concrete example of what I mean:
I hardly ever code in JavaScript; I'm mostly a backend developer. But sometimes I want to quickly fix a problem with our frontend that's preventing end-to-end testing; or I want to add a proof-of-concept frontend half to a new backend feature, to demonstrate to the frontend devs by example the way the frontend should be using the new API endpoint.
Now, I can sit down with a JS syntax + browser-DOM API cheat-sheet, and probably, eventually write correct code that doesn't accidentally e.g. incorrectly reject reject zero or empty strings because they're "false-y", or incorrectly interpolate the literal string "null" into a template string, or incorrectly try to call Element.setAttribute with a boolean true instead of an empty string (or any of JS's other thousand warts.) And I can do that because I have written some JS, and have been bitten by those things, just enough times now to recognize those JS code smells when I see them when reviewing code.
But just because I can recognize bad JS code, doesn't mean that I can instantly conjure to mind whole blocks of JS code that do everything right and avoid all those pitfalls. I know "the right way" exists, and I've probably even used it before, and I would know it if I saw it... but it's not "on the tip of my tongue" like it would be for languages I'm more familiar with. I'd probably need to look it up, or check-and-test in a REPL, or look at some other code in the codebase to verify how it's done.
With an LLM, though, I can just tell it the pseudocode (or equivalent code in a language I know better), get an initial attempt at the JS version of it out, immediately see whether it passes the "sniff test"; and if it doesn't, iterate just by pointing out my concerns in plain English — which will either result in code updated to solve the problem, or an explanation of why my concern isn't relevant. (Which, in the latter case, is a learning opportunity — but one to follow up in non-LLM sources.)
The product of this iteration process is basically the same JS code I would have written myself — the same code I wanted to write myself, but didn't remember exactly "how it went." But I didn't have to spend any time dredging my memory for "how it went." The LLM handled that part.
I would liken this to the difference between asking someone who knows anatomy but only ever does sculpture, to draw (rather than sculpt) someone's face; vs sitting the sculptor in front of a professional illustrator (who also knows anatomy), and having the sculptor describe the person's face to the illustrator in anatomical terms, with the sketch being iteratively improved through conversation and observation. The illustrator won't perfectly understand the requirements of the sculptor immediately — but the illustrator is still a lot more fluent in the medium than the sculptor is; and both parties have all the required knowledge of the domain (anatomy) to communicate efficiently about the sculptor's vision. So it still goes faster!
(Or if you’re being paid to waste time, maybe consider coding in assembly?)
So don’t be afraid. Learn to use the tools. They’re not magic, so stop expecting that. It’s like anything else, good at some things and not others.
They don't have high status even today, imagine in a world where they will be seen as just reviewers for AI code...
Try putting on a dating website that you work at Google vs you work in agriculture and tell us which yielded more dates.
With so many hits, it's about hitting all the checkmarks instead of minmaxing on one check.
My comment was more a critique on online dating culture and the values it weighs compared to in person meetups.
I think revealing the domain each programmer works in and asking in hose domains would reveal obvious trends. I imagine if you work in Web that you'll get workable enough AI gen code, but something like High Performance computing would get slop worse than copying and lasting the first result on Stackoverflow.
A model is only as good as its learning set, and not all types are code are readily able to be indexable.
How people deal with this is they start by writing the test case.
Once they have that, debugging that 25% comes relatively easily and after that its basically packaging up the PR
If anything that's probably an underestimate. Not to downplay the complexity in much of what Google does but I'm sure they also do an absolute ton of tedious, boring CRUD operations that an AI could write.
Their internal AI tools are presumably trained on their code, and it wouldn't surprise me if the AI is capable of much more internally than public coding AIs are.
maybe the ai generates 100% of the company's new code, and then by the time the programmers have fixed it, only 25% is left of the AI's ship of Theseus
There's even a buzzword for it: KTLO (keep the lights on). You don't want to be spending 100% of your time on KTLO work, but it's unrealistic to expect to do done of it. Most software engineers would gladly outsource this type of scutwork.
Some places also call this "RTB" for "run the business" type work. Nothing but respect for the engineers who enjoy that kind of approach, I work with several!
> how trivial the problems they solve are
A single line of code IS trivial. Simple code is good code. If I write the first 3 lines of a complex method and I let Copilot complete the 4th, that's 25% of my code written by an LLM.
These tools have exploded in popularity for good reason. If they were no good, people wouldn't be using them.
I can only assume people making such comments don't actually code on a daily basis and use these tools daily. Either that or you haven't figured out the knack of how to make it work properly for you.
With c++ my experience is that the results are completely worthless. It saves you from writing a few keywords but nothing that really helps in a big way.
Yes Copilot CAN work, for example writing some JS or filter functions, but in my job these trivial snippets are rather uncommon.
I‘d genuinely love to see some resources that show its usefulness that arent just PR bs.
What does that even mean? What are you expecting to see?
I've seen people who can't code ship entire new applications which actually work, in a few days or so. That to me seems more productive?
I use these tools daily in a FAANG level SWE role and they help me debug issues quickly - all the time, especially with tech I'm new to and have no experience with. I really don't understand the hate - it's like skipping stack overflow and giving you the ideal answer a lot faster.
Nobody likes to shout that they're using these tools but most people are.
You're saying anything that's ever been popular is popular for a good reason? You can't think of counter examples that disprove this?
You're saying anything that people decide to do is good, or else people wouldn't do it? People never act irrationally? People never blindly act on trends? People never sacrifice long-term results for short-term gain? You can't come up with any counter examples?
I use these tools daily and they help me immensely. If you prefer Googling for docs, browsing stack overflow, or even flicking through textbooks to find the answers/materials you need - that's great! Do what works for you. I value my time slightly more than that and prefer not to remain stuck in the past.
Perhaps you hold all the information you need in your head like an oracle and never need to learn new concepts or ever forget syntax? Wonderful. The rest of us aren't so naturally talented, so have found these new tools super helpful.
Except of course AI at least can do spelling. (Or at least I haven't encountered a problem in that regard.)
I'm highly skeptical regarding LLM-assisted development. But I must admit: it works. If paired with an experienced senior developer. IMHO it must not be used otherwise.
Damn, that’s a good way of putting it. But I’ll go one further:
replace "AI" with "junior dev who doesn’t like reading documentation or googling how things work so instead confidently types away while guessing the syntax and API so it kind of looks right”
Currently, they don't learn skills as fast as a motivated intern. A stellar intern can go from no idea to "makes relevant contributions to our product with significant independence and low error rate" (hi Matt if you ever see this) in 3 months. LLMs, to my understanding, take significantly more attention from super smart people working long hours and an army of mechanical Turks, but won't be able to independently implement a feature and will still have a higher error rate in the same 3 months.
It's still super impressive what LLMs can do, but that same intern is going to keep growing at that faster rate in skills and competency as they go from jr->mid->sr. Sure the intern won't have as large of a competency pool, and takes longer to respond to any given question, but the scope of what they can implement is so much greater.
Surely yes.
I (not at Google) rarely use the LLM for anything more than two lines at a time, but it writes/autocompletes 25% of my code no problem.
I believe Google have character-level telemetry for measuring things like this, so they can easily count it in a way that can be called "writing 25% of the code".
Having plenty of "trivial code" isn't an indictment of the organisation. Every codebase has parts that are straightforward.
For the most part, it drives itself.
Yes, the majority of my code is trivial. But I've also had ai iterate on some very non trivial work including writing the test suite.
It's basically autocomplete on steroids that predicts your next change in the file, not just the next change on the line.
The copy paste from stack overflow trope is a bit weird, I haven't done that in ten years and I don't think the code it produces is that low quality either. Copy paste from an open source repo on GitHub maybe?
For most of my career, Software Engineering was a misnomer. The field was too young, and the tools used changed too quickly, for an appreciable amount of the work to be systematic and boring enough to consider it an Engineering discipline.
I think we're now at the point where Software Engineering is... actually Engineering. Particularly in the case of large established companies that take software seriously, like Google (as opposed to e.g. a bank).
Call it "trivial" and "boring" all you want, but at some point a road is just a road, and a train track is just a train track, and if it's not "trivial and boring" then you've probably fucked up pretty badly.
I’m an engineer who writes code since 20 years and it’s far away from trivial . Maybe to do web dev for a simple Webshop is. Elsewhere software has often times special requirements. Be them technical or domain wise both make the process complex and not simple IMHO
Not all engineering is boring. Also, boring is not bad.
A lot of my career has been spent working to make software boring. To the extent that I've helped contribute to the status quo, where we can build certain types of software in a relatively secure fashion and on relatively predictable timelines, I am proud to have made the world more boring!
(Also, complexity can be extraordinarily boring. Some of the most complex things are also the most boring. Nothing more boring than a set of business rules that has an irreducible complexity coming in at 5,211 lines of if-else blocks wrapped in two while loops! Give me a simple set of partial differential equations any day -- much more exciting to work with those! If you're the type of person who enjoys reading tax code, then we just have different definitions of boring; and if you're the type of person doesn't think tax code is complex, then I'm just a dummy compared to you :))
But e.g. in the early naughts doing structural engineering work for residential new build projects was certainly less engaging and exciting work than building websites.
Most engineering works aims for repeatable and predictable outcomes. That's a good thing, and it's not easy to achieve! But if Software has reached the point where the process of building certain types of software is "repeatable and predictable", and if Google needs a lot of that type of software, then if the main criticism of AI code assistants is "it's only good for repeatable and predictable", well, then the criticism isn't exactly the indictment that skeptics think it is.
There is nothing wrong with boring in the sense I'm using it. Boring can be tremendously intellectually demanding. Also, predictable and repeatable processes are incredibly important if you want quality work at scale. Engineering is a good thing. Maturing as a field is a good thing.
But if we're maturing from "wild west everything is a greenfield project" to "70% of things are pretty systematic and repeatable" then that says something about the criticism of AI coding assistants as being only good for the systematic and repeatable stuff, right?
Also: the AI coding assistant paradigm is coming for structural/mechanical/civil engineering next, and in a big way!
there is a 3rd possibility as well: having spent a huge chunk of change on these techniques, why not overhype it (not outright lie about it) and hope to, somewhat recoup the cost from unsuspecting masses ?
Thankfully I don't find it subtle but plain wrong for anything but trivial stuff. I use it (and pay an AI subscription) for things where false positive won't ruin the day, like parameters validation.
But for anything advanced, it's pretty hopeless.
I've talked with lawyers: same thing. With doctors: same thing.
Which ain't no surprise see how these things do work.
> Like, isn't this announcement a terrible indictment of how inexperienced their engineers are, or how trivial the problems they solve are, or both?
Probably lots of highly repetitive boilerplate stuff everywhere. Which in itself is quite horrifying if you think about it.
Source: I work there, see my previous comment.
I'm looking for a new job, so I've been grinding leetcode (oof). I'm an experienced engineer and have worked at multiple FAANGs, so I'm pretty good at leetcode.
Today I solved a leetcode problem 95% of the way to completion, but there was a subtle bug (maybe 10% of the test cases failing). I decided to see if Claude could help debug the code.
I put the problem and the code into Claude and asked it to debug. Over the course of the conversation, Claude managed to provide 5 or 6 totally plausible but also completely wrong "fixes". Luckily, I am experienced enough at leetcode, and leetcode problems are simple enough, that I could easily tell that Claude was mistaken. Note that I am also very experienced with prompt engineering, as I ran a startup that used prompt engineering very heavily. Maybe it's a skill issue (my company did fail, hence why I need a job), but somehow I doubt it.
Eventually, I found the bug on my own, without Claude's help. But leetcode are super simple, with known answers, and probably mostly in the training set! I can't imagine writing a big system and using an LLM heavily.
Similarly, the other day I was trying to learn about e-graphs (the data structure). I went to Claude for help. I noticed that the more I used Claude, the more confused I became. I found other sources, and as it turns out, Claude was subtly wrong about e-graphs, an uncommon but reasonably well-researched data structure! Once again, it's lucky I was able to recognize that something was up. If the problem wasn't limited in scope, I'd have been totally lost!
I use LLMs to help me code. I'm pro new technology. But when I see people bragging on Twitter about their fully automated coding solutions, or coding complex systems, or using LLMs for medical records or law or military or other highly critical domains, I seriously question their wisdom and experience.
Or maybe there's a KPI around lines of code or commits.
It is not Netflix or Airbnb or Stripe etc making this claim, google managers have a vested interest in this.
If this metric was meaningful either of two things should have happened - google should have fired 25 % developers or built 25 % more product .
Both of this would visible in their financial reporting and has not happened.
metrics like this claim depends on how you count, that is easily gamed and can be made to show any % between 0-99 you want. Of the top of head
- I could count all AI generated code used for training as new code
- consider compiler output to assembly as AI code by adding some meaningless AI step in it
- code generated with boilerplate perhaps even generated by llm now
- mix autocomplete with llm prompts so on
The number only needs to believable , 25 is believable now, it is not true but you would believe it >50 has psychological significance and bad PR on machines replacing humans jobs , less than 10 is bad for AI sales , 25 works all the commenters in this thread is testament to that
This is not a bad thing since you can improve, but constantly dismissing something that a lot of people are finding an amazing productivity boost should give you some pause.
But as of now the field is full of swamps. Of grifters, of people with a solution looking for a problem. Of outright scams of questionable legality being challenged as we speak.
I'll wait until the swamps work itself out before evaluating an LLM workflow.
LLMs are being used right now by a lot of people, myself included, to do tasks which we would have never bothered with before.
Again, if you don't know how to use them you can learn.
It's a legal nightmare in my domain as of now, so I'll make sure the Sam Breaker-Friends are weeded out. If it's really all the hype it won't be going anywhere in 5 years.
In a purely technical vacuum though: it is truly amazing tech. I will give it that. Although it both excites and alarms me that apparently the power output predicted to properly leverage this at scale is having tech companies consider an investment in nuclear power.
I won't blame people for that latter, I'd love a good quick way out of traditional work as well (gives me more time to hack on stuff without money troubles). But it's not a good model for curiosity and scrutiny. Again, I'll wait it out. Take care.
Investing trillions in carbon free energy for AI is the most benign form of bubble I can imagine. If the bubble pops we have enough base load for the next century and don't die from climate change. If it doesn't we have the expertise to keep building large nuclear power plants.
I think you underestimate the amount of boiler-plate code that a typical job at Google requires. I found it soul-crushingly boring (though their pay is insane).
Well, Rob Pike said same thing about experience and that seemed to pissed lot of people endlessly.
However I don't think it as indictment It just seems very reasonable to me. In fact 25% seem to be on lower end. Amazon seems to have thousands of software engineers who are doing API calling API calling API.. kind of crap. Now their annual income might be more than my lifetime earnings. But to think that all these highly paid engineers are doing highly complex work that need high skills seems just a myth that is useful to boost ego of engineers and their employers alike.
That's not that much...
Not that I'm really discounting the value of AI here. For example, I've found a ton of value and saved time getting AI to write CDKTF (basically, Terraform in Typescript) config scripts for me. I don't write Terraform that often, there are a ton of options I always forget, etc. So asking ChatGPT to write a Terraform config for, say, a new scheduled task for example saves me from a lot of manual lookup.
But at the same time, the AI isn't really writing the complicated logic pieces for me. I think that comes down to the fact that when I do need to write complicated logic, I'm a decent enough programmer that it's probably faster for me to write it out in a high-level programming language than write it in English first.
IMO it's only really an issue if a competent human wasn't involved in the process, basically a person who could have written it if needed, then they do the work connecting it to the useful stuff, and have appropriate QA/testing in place...the latter often taking far more effort than the actual writing-the-code time itself, even when a human does it.
That said, I've seen even higher ratios. But never in any place that survived for long.
Everything is getting forced into a scalable, general purpose way, that most apps have to add a ridiculous amount of boilerplate.
Combine that with generic functions, framework boilerplate, OS/browser stuff, or explicit x-y-z code then your 'boilerplate' (ie repetitive, easily reproducible) easily gets to 25% of code you're programmers write every month. If your job is >75% pure human cognition problem solving you're probably in a higher tier of jobs than the vast majority of programmers on the planet.
It's only boilerplate if you write it again to set almost the same thing again. What, granted, if you are writing bare terraform config, it's probably both.
But on either case, if your terraform config is repetitive and a large part of the code on an entire thing (not a repo, repos are arbitraty divisions, maybe "product", but it's also a bad name). Than that thing is certainly close to useless.
Architecturally, it sounds like different architecture components map somewhere close to 1:1 to teams, rather than teams hacking components to be closer coupled to each other because they have the same ownership.
I'd see too much boilerplate as being a organization/management org issue rather than a code architecture issue
Not the developer who has written the same effective stanza 10 times before.
There are still things I do in my IDE that I can’t seem to get AI to do. It’s not really close yet. I don’t doubt it could get there eventually, but I suppose I don’t believe it’s about to eat those parts of the industry.
I do anticipate a massive issue from lower skill software jobs vanishing. I don’t know what entry into the industry will look like. There will be a strange gap that’s filled by AI and some people who use it to do basic things but have no idea how it does it. They will be somewhat like data entry workers, knowing how to use a spreadsheet or word processor but having no idea how the program actually works let alone the underlying operating system. I fully expect that to happen, and I can’t properly imagine what the implications will be.
With g3's immense amount of context, LLMs can vastly help you discover how other people are using existing libraries.
in regards to how others are using libraries, that’s where the technology will excel— re-writing code. once it has a stable AST to work with, the mathematical equation it is solving is a refactor.
until it has that AST that solves the business need, the game is just prompt spaghetti until it hits altitude to be able to refactor.
Google could be writing the same amount of code with fewer developers (they have had multiple layoffs lately), or their developers could be focusing more of their time and attention on the code they do write.
The truth is likely somewhere in between.
That may explain why google search has, in the past couple of months, become so unusable for me that I switched (happily) to kagi.
> New tool bypasses Google Chrome’s new cookie encryption system
- You know which 20K lines need changing - You have perfect QA - Nothing ever goes wrong in deployment.
I think there's a tendency in our industry to only take the hypotenuse of curves at the steepest point
I consider myself risk-averse and even I am contemplating starting a small business in the event I get laid off.
New successful businesses are being created all the time. We just focus on the ones that have already been successful for a long time.
All of that said, there are a lot of products that are produced by large companies and are just bad. Don’t be afraid to go after a Goliath if you see an opportunity.
It really isn't. Even if you get laid off from a large tech company, you probably didn't have to pay a cent to get the job there in the first place, and you started drawing a paycheck right away (after the initial delay due to the pay cycle). If you only work there for 6 months, you can save a really good amount of money if you have frugal habits.
Starting a company isn't nearly as easy, usually requires up-front investment, and there can be a long time before you generate any profit. Either you need some business idea that's going to generate profit (or at least enough revenue to give the founder(s) a paycheck), or a business loan or other funding, which means convincing someone to invest in your company somehow.
Starting your own company only sounds appealing if you ignore reality, or have the privilege of having plenty of cash saved up for such a venture.
If I was working a typical corp job, I would "quiet quit" and start using my excess savings to run small experiments. Maybe run like 3 in parallel a month and let them cook.
Example - start a niche blog based on my hobby, hire 3 writers and pay them up to $3k a month to write content for the blog. Let it cook for months. If it gets traffic, monetize with ads. If very profitable, quit job.
Starting an internet based business if fairly cheap nowadays, especially with cursor and ai.
it's like companies paying all those todolist and tutorial apps left running on aws ec2 instances in 2007ish.
I'd be worried if i were a google investor. lol.