Did GitHub Copilot increase my productivity?
trace.yshui.dev
trace.yshui.dev
Long analogy short, this is where I think AI for coding is now. It gets things wrong enough that I have to manually check everything it does and correct it, to the point where I might as well just do it myself in the first place. This might not always be the case, but that's where I feel it is right now.
[0] Entity Framework has moved on a lot since then, and apparently now can be trusted to lazily load data. I don't know because...
[1] I spat the dummy, replaced Windows with Linux, and started learning Go. Which does exactly what it says it does, with no magic. Exactly what I needed, and I still love Go for this.
What I have learned over the years is that the only way to properly use ORM is as a fancy query tool. Build the query, fetch/update data, MOVE THE DATA to separate business objects. Don't leave ORM entities shared across the sea of objects!
Phew, thanks, I got that off my chest.
I hate EF and everything it stands for. :)
[edit] for instance, I have a case where I use a few dozen custom queries on timers to trawl through massive live data and reduce it into a separate analytics DB. Using everything from window functions to cron timers to janky PHP code that just slams results from separate DBs together to provide relatively accurate real-time results. At the end from that drastically reduced set in the analytics DB... sure, I'm happy to let the client summarize whatever they want with Metabase. But those things just couldn't be done with an ORM, and why would I want to?
This is compatible with the OC suggestion of using ORMs as a "fancy query builder" and nothing more, which I strongly support.
(I would say that it is conceptually perverse for a client of a system to have authority over it. Specifically, for a database client to define its schema.)
This is where migrations also become sort of annoying. Because if you use them. Then it is harder to fix the mistakes since you can’t just change your DB without using the ORM or you’ll typically break your migration stream or at least run into a lot of troubles with it.
And what is the plus side of having a code-first DB really? You can fairly easily store those “alter table” changes as you go along and have full availability of history in a very readable way that anyone, including people not using C#, Java, Python.
Which is the other issue with ORMs. If you have multiple consumers of your data. Then an ORM most likely won’t consider that as it alters your “models”.
For a lot of projects this is a non-issue, especially at first. Then 10 years down the line, it becomes a full blown nightmare and you eventually stop using the ORM. After spending a lot of resources cleaning up your technical debt.
The benefits should be obvious if you've used ORMs. They are an object that represents your database data in code rather than in a table where you can't touch it. If you have code that brings data from a database into code, congratulations, you've implemented part of an ORM. Having the data model defined "in code" treats the code as first-class instead of the SQL, which makes sense from an ergonomics perspective, you will spend much more time with the code objects than you will the SQL schemas. Either way, you will have two versions: a SQL version and a code version. You might as well get both from writing one.
If you can read alter table in SQL, you can probably read migrations.AddField in Python, and whatever the equivalent is in the other languages. I still am waiting with bated breath for the problems with much maligned (by some) ORMs to arrive.
I’m not really against ORMs as such. I’m not a fan of code-first databases for anything serious, but as far as CRUD operations goes I don’t see why you wouldn’t use an ORM until it fails you, which it won’t in most cases, and in those cases where it does… well similar to what I said earlier you just wouldn’t have build it to scale from the beginning anyway, and if you had and it turned out it didn’t need to scale then you probably wasted a lot of developer resources to do so.
- Proper DB design first. You should be able to remove the ORM and DB should still function as intended. This means application-side cascade operations or application-side inheritance is banned.
- No entities with magical collections pointing to each other. In other words, no n to n relations handled by ORM layer. Create in-between table, for gods sake. Otherwise it becomes incredibly confusing and barely maintainable.
- Prefer fetching data in a way that does not populate collections. In other words, fetch the most fine-grained entity and join related data. Best if you craft special record entities to fetch data into (easy with EF or Doctrine).
- Most ORMs allow you to inspect what kind of queries you create. Use it as query building tool. Inspect queries often, don't do insane join chains and other silly stuff.
I would use ORM in one kind of app: where I would work with data that shares records that might need to be either inserted or updated, and there is several nesting levels of this kind of fun. You know, you need to either insert or update entity, if it exists, you should update, and then assign related entities to it, if it does not, then you should insert, and assign related entities to the newly created id. The ORM can easily deal with that, and on top of that it can do efficient batched queries, which would be really annoying and error-prone to hand-craft.
If the app does not require this kind of database with these kind of relations, I would not use ORM.
So, I have a database that looks like this. My method was to lay out the database myself, by hand, and then use EF's facility to generate EF code from an existing database. The bridge table was recognized as being nothing but the embodiment of a many-to-many relation and the EF code autogenerated the collections you don't like.
Is this a problem? If you do things the other way around, the ORM creates the same table and it's still there in the database. It isn't possible not to create the bridge table. Why is that case different?
I just fail to see what else would you do, besides implementing a bug-ridden, half-ORM yourself.
1. Very often we need generated SQL because writing SQL for primitive CRUD operations is hell tedious and error-prone (as well as writing UI forms connected to these CRUD endpoints, so I prefer to generate them too).
2. Structured Query Language being very poorly structured is indeed a huge resource drain when developing and maintaining complex queries. PRQL and the like try to address this, but that's an entirely different level of abstraction.
3. Unfortunately, when efficiency matters we have to resort to writing hand-optimized SQL. And this usually happens exactly when we terribly need a well-composing query language.
But that comes to the second problem. Most devs I meet seem to be deathly allergic to SQL. :)
One project I had a dev come to me asking me to look at a bug in the thing. Having never seen that particular ORM before I was able to diagnose what was wrong. Because MS ORMs have the same issues over and over (going back to the 90s). You better read those docs! Because whatever they did in this stack will be in their next one when they abandon it in place 3 years from now.
Depends on the ORM.
I have noticed that typically, 'unit of work' type ORMs (EFCore and Hibernate/NHibernate as examples) prevent being 'true to the ORM' but 'efficient'.
i.e. Hibernate and EFCore (pre 7 or 8.0ish) cannot do a 'single pass update'. You have to first pull the entities in, and it does a per-entity-id update statement.
> I just fail to see what else would you do, besides implementing a bug-ridden, half-ORM yourself.
Eh, you can do 'basic' active-record style builders on top of dapper as an afternoon kata, if you keep feature set simple, shouldn't have bugs.
That said, I prefer micro-ORMs that at most provide a DSL for the SQL layer. less surprises and more concise code.
Traditionally, though, SQL generation was known as query building. The query was executed via database engine or database driver, depending on the particulars. ORM, as the name originally implied, was the step that converted the relations into structured objects (and vice versa). So, yes, technically if you maintain your data as tuples end to end you are not utilizing ORM. Lastly, there once was what was known as the active record pattern that tried to combine all of these distinct features into some kind of unified feature set.
But we're in a new age. Tradition has gone out the window. Computing terms have no consistency to speak of, and not just when it comes to databases. Indeed, most people will call any kind of database-related code ORM these days. It's just funny that ORM no longer means object-relational mapping.
Object-relational mapping was originally coined in the Smalltalk world, so objects were in front of mind, but it was really about type conversion. I am not sure that functions or methods are significant. It may be reasonable to say that a struct is an object, for all intents and purposes.
A pendant might say that what flimsy definition Kay did give for object-oriented programming was just a laundry list of Smalltalk features, meaning that Smalltalk is (probably) the only object-oriented language out there, and therefore ORM can only exist within the Smalltalk ecosystem. But I'm not sure tradition ever latched onto that, perhaps in large part because Kay didn't do a good job of articulating himself.
ORMs create this false equivalence (and in this sense, so does Django's admin interface despite using tuples instead of classes). I can see the sense of this, vaguely, for an admin interface, but it's still a false equivalence.
Simple: because if I don't, I'm going to spend the rest of my career explaining why I didn't to people extremely skeptical of that decision. Meanwhile even people like me tend to just shrug and quietly go "oh, an ORM? Well, that's the price of doing the job."
Also, ORMs are an endless source of well-paid jobs for people who actually learned relational algebra at some point in their lives, and that's not a compliment to ORMs.
On top of that most ORMs have migrations, connection management, transaction management, schema management and type-generation built-in.
Some ORMs have inherently bad design choices though, like lazy loading or implicit transaction sharing between different parts of the code. Most modern ORMs don't really have that stuff anymore.
But in .Net, EF is still the most common way of accessing data (I have heard, because I stopped using it over a decade ago).
its sql in elixir syntax with a bunch of QOL improvements.
for one thing, I can seperate my subqueries into separate variables
``` sub_q = from(l in Like) |> where([l], l.user_id == ^user_id) |> select([l], %{ user_id: l.user_id, likes_count: count(l.id) }) |> group_by([l], l.user_id)
main_query = from(u in User) |> join(:left, [u], likes_count in ^subquery(sub_q), on: likes_count.user_id == u.id, as: :likes_count) |> select([u, likes_count: l], %{ name: u.name, likes: l.likes_count, }) |> where([u], u.id == ^user_id)
user = main_query |> Repo.one()
```
Being able to think directly in sql lets you perform optimal queries once you understand sql. and imho, this much cleaner than tha equivalen sql to write. it also takes care of input sanatization and bindings.
Then I learned Ecto/Phoenix and this is truly the best way. Ecto is so close and translateable to raw SQL that there's little to no friction added by it, but it handles all the stuff you don't want to have to do by hand (like query building, parameterization, etc). Ecto is a real breath of fresh air and I find myself writing quick scripts that hit a database in Elixir just so I can use Ecto! I also love how easy Ecto makes it to model database tables that were originally defined by another language/framework or even by hand. Trying to do that with ActiveRecord or another ORM is usually a recipe for extreme pain, but with Ecto it's so easy.
It’s so elegant and the Lego blocks (query, schema, change set, repo) can be mixed and matched in different ways.
I’ve even used schemas and change sets to validate API requests and trivially provide very nice, clean, and specific errors, while getting perfectly typed structs when things are validated.
As an alternative to query-by-example (QBE)?
Too little abstraction is just as bad as too much.
Gee, I wonder if there's a better way to do so that is not such a lazy job. But doing it properly, like .NET does, is supposedly too much effort!
I would not be enthusiastic about a system where I receive database query results for review, before delivering them to an end user somewhere on this planet. However, I am more than happy to get some extra help in communicating code from my brain to a compiler.
Good developers know to appreciate that and wouldn’t want to touch Go ecosystem afterwards with a 10 feet pole.
So it still seems flakey. I've never worked a single job that chose EF that didn't end up regretting it. Either from it being unreliable, migration hell or awful performance.
"It allows you to treat your database like an in-memory enumerable"
Then devs go and do exactly that and wonder why performance is so terrible...
I hate EF.
We had an issue last week where we had an obect like
public class Foo
{
public List<Bar> Bars { get; set; }
}
await _dbContext.Foos.ToListAsync();
No errors, no exceptions, just empty lists where we'd expect data.
And so often we have to debug and see what SQL its actually generating, then spend time trying to get it to generate reasonable sql, when if we were using sprocs we could just write the damn sql quicker.
Another issue we have is the _EFMIgratoinsHistory table.
Sometimes we will deploy and get a load of migration errors, as it tries to run migrations its already ran... SO they all fail and then the API doesn't come back up... The fix ? TUrn it off and on again and it stops trying to re-run migrations its already ran!
Assuming you haven't missed to add an .Include[0], please consider submitting an issue(s) to https://github.com/dotnet/efcore
[0] https://learn.microsoft.com/en-us/ef/core/querying/related-d...
EF tries to be smart and will fix up the property in memory if the referenced entities are returned in separate queries. but if those queries don't return all records in `Foo.Bar`, `Foo.Bar` will only be partially populated.
this can be confusing and is one of the reasons i almost never use navigation properties when working with EF.
e.g. stick a breakpoint, step over, see in the debugger that it was not populating everything it should. Then run it again, do the same and see different results. Exact same code, exact same db, different results.
5000 results back from the db, anything between 5000 and a handful were only fully correctly populated.
There is also the AutoIncludeAttribute that you can specify on entity fields directly to always include those fields for every query.
My main complaints with EF are that the scaffolding and migration commands for the CLI tool are nearly impossible to debug if they error during the run.
But when they run right, they save me a ton of time in managing schema changes. Honestly, I consider that part worth all the rest.
There can also be some difficulty getting queries right when there are cyclical references. Filtering "parent" entities based on "child" ones in a single query can also be difficult, and also can't be composed with Expression callbacks.
But in any difficult case, I can always fall back on ADO.NET with a manual query (there are also ways of injecting manual query bits into EF queries). Which is what we'd be doing without EF, so I don't get the complaints about EF "getting in the way".
I have had personal experience with this. And seen others telling me as well. These AI things often suggest wrong code, or even with bugs. If you begin your work by assuming AI is suggesting the correct code you can go hours, to even days debugging things in the wrong place. Secondly when you do arrive at a place where you find the bug in the AI generated code, it can't seem to fix or even modify it, because it misses context in which the which itself generated at the first place. Thirdly the AI itself can interpret your questions in a way you didn't mean.
As of now AI generated code is not for serious work of any kind.
My guess a whole new paradigm of programming is needed, where you will more or less talk to AI in a programming language itself, some what like lisp. I mean a proper programming language at a very abstract level, which can be interpreted in only one possible meaning, and hence not subject to interpretation.
Code generation is quite old though, and also quite common, also outside the Lisp-family. When doing non-trivial systems development in Java you tend to use it a lot, especially with XML as an intermediary, abstracted language.
To some degree. If you're using it for anything serious you're still going to help it along a lot. It's rather easy to do so, however, and I certainly wouldn't consider writing your own code as fast or easy as simply telling EF how you want it to do certain things.
I'm not an overall fan of EF. I especially dislike how it's model builder does not share interoperability with other .Net libraries which also use it. I also don't really like the magic .Net does behind the scenes. EF as a whole has been one of the better ORMs for any language since .net core. I'd still personally much prefer something like Rust's diesel, but whenever I have to work with C# I tend to also use EF.
>Diesel: A safe, extensible ORM and Query Builder for Rust
FWIW, it can actually work 'on top' of an Existing EF Core context+mappings (i.e. if you have an existing project and want to migrate, or just need the better feature-set for a specific case.) or you can get pretty close to 'yolo by convention' depending on case. In general though it's a lot less ceremony to start messing around.
Thing is, reading code is way faster than writing code
When I'm not sure in the someones code I have to double or triple check it be sure that I understand it correctly and to verify that there no somehow hidden missed steps or side effects.
Just means I now spend a lot of time rewriting it which I could have just done in the first place but now I’ve wasted time asking GPT too.
Even if that were true, reading code is reasonably faster than typing it out and then reading it again to check it.
This might be dependant on the programming language, some languages are way more popular and have way more questions on StackOverflow and Reddit and repos on github, so the answers will be better.
When I use copilot for JS it's right 90% of the time.
And where it's 'wrong' it's usually just stuff it skipped over because it didnt have proper context.
Also if you relied on lazy loading properties after the DbContext had been disposed (after the using() block) you were out of luck.
With old EF we would turn off lazy loading to make sure devs always got an exception if they hadn’t used .Include() to bring the related entities back in their initial query. Querying the database should always be explicit not lurking behind property getters.
Fortunately with EF core MS realized this and it’s off by default. EF with wise use of .Include and no lazy loading is a pretty good ORM!
I strongly believe that software we develop should feel like magic to the users
The tools that we use to build them should not
The good way to do this is to write good unit tests for the code.
Which we should be doing anyway!
oh god, I have used Java with Hibernate a lot and once I read "Lazy Loading" I didn't even need to finish reading the post.
One thing I've found about Copilot is that it introduces me to novel ways to solve problems and more obscure language features. It makes me a better coder because I'm constantly learning. But do I want to be spending my time learning or do I want to make that deadline that's coming up?
My problem is running into it's limitations, mostly around resources. I have tried giving it larger tasks and it takes bloody forever.
"Given this unstructured data, create CSV output for all platforms, with each line containing the manual, and model, ignoring the text in parenthesis."
Works great except for God-awful performance and stopping half way through. I had to break out each section and paste it into the prompt and let it work on small pieces. We need to get to the next level with this, especially for paying customers.
More concerning is that I see a clear pattern in smaller companies of hiring seniors and turning them loose with AI assistants instead of hiring junior devs. The prospect is attractive to nearly every stakeholder and the propensity to put off hiring "until next quarter" in light of this is a constant siren song. There is a lot of gravity pulling in this direction with the short-term thinking and distractions that are thoroughly soaked into the business world these days. Supposedly, one third of Gen Z (20-25 yrs old) are sitting at home, up from 22% in 1990.
I'm one of those seniors happily putting off hiring, but I find the situation and it's wider impact on the future very unnerving.
A lot of the discrepancy between people's experiences is simply due to the fact there's there's a massive range of programming complexity/difficulty that people can be trying to apply AI to. If your programming is mostly lower complexity stuff, non-critical stuff, or simply defined stuff, it obviously works better.
I try to use AI when I get stuck on a hard problem/algorithm, hoping that it can provide an answer/solution to unblock me. But when I'm stuck the problem I'm facing is so complicated that there's no chance at all that AI is actually going to be able to help me with it. I see absolutely no point in using AI when I already know how to solve a problem, I just solve it. I only turn to it when I need help, and it can never help me.
For your analogy to hold, what is the impedance mismatch between programming and Copilot?
It's a different story for test code. Test code is often formulaic and "standardized" (given/when/then). For instance, I find myself writing the first test case and Copilot can come up with additional test cases. Or I might write the method name ( FeatureUnderTest_Scenario_ExpectedOutcome) and Copilot provides the implementation.
I have not found any value in Copilot chat.
Then there is the design side of things. I really feel bad for designers of Icons now that you can get some really good one really fast by tasking one of the image generating AIs.
I’m not sure LLMs will ever really be capable helpers as far as programming goes. Well I guess it’s two part, they can help with trivial tasks, but they can’t help with anything related to the actual work of generating business value with code. It’s two-sided of course. They certainly allow a lot of people write functioning, though really shitty, code. Which is a huge benefit for a lot of programming tasks where it doesn’t really matter that it’s inefficient and well terrible. We’ve already seen our more digitally inclined employees make great things with power apps, most of which are eventually replaced by more robust software as they scale. But we also see small Python programs helping out with tiny personal tasks around our offices, and while IT operations aren’t too happy it’s generating a lot of individual value that wasn’t there before.
So pair it with mutation testing!
It wrote a python app. It hard coded the API key and the CSV file. And then it told me to pass the file name as an argument. lol.
I just asked it to fix that and tested with a two line csv. Worked like a charm and saved me quite a bit of time trying to figure a few new things out.
But a proper programmer would have been slowed down by this, for sure.
As someone who spends 80% of his time in the backend, I find it great for JavaScript whereas it's not so good for Django which I know pretty well.It can still be useful though and is often faster than looking up docs for specific things.
Just churning out more near copies is not a good answer.
It doesn't really matter how many helper functions you extract from your test code, in the end you have to string them together and then make assertions, and that part will always be repetitive and formulaic. If you've extracted a lot of shared code, then it might look something like "do this high-level business thing and then check that this other high-level business thing is true". But that is still going to need to be written a dozen times to cover all the test cases, and you're still going to want test names that match the test content.
There's a certain amount of repetition and formulaism that will never go away and that copilot is very good at.
1. It can make it harder to see what's actually being tested if there are too many layers of abstraction in the test.
2. Complex test code can have significant bugs of its own that can result in false passes. What tests the test code?
Thus I generally see repetitive or copy/pasted test code as a necessary evil a lot of the time.
I now find it is best to be very explicit in the individual test code about what the conditions are of that specific test.
I could just leave the discussion I guess, but in the interest of discourse, I don't find your statement meaningful because we're not all working in languages that I think you refer to. Our unit tests are absolutely not perfect and don't offer perfect guarantees, as we're fallible and will write fallible code.
And as such, I just don't understand what point you're trying to make by saying that LLM generated tests are no good because they can't offer perfect guarantees.
I’m on mobile so it’s hard to reference what I previously said but I’m assuming my statement needs to be weakened a bit to be correct. What I meant to say was that unit tests provide essentially no value because they can’t offer perfect guarantees, which is probably different than what I originally said. I’m assuming I just said “they offer no value” which is probably false in some cases for some people and some teams depending on their definition of value. My point was that unit tests do not make sense insofar as their purpose is to provide guarantees about the behavior of code because the information they provide does not meet the standard definition of “a guarantee”. For the above mentioned people/teams/situations/value definitions, they may make sense.
Hope that clarifies what I was trying to say.
Regarding languages, algebraic structures can be implemented in any Turing complete language. Likewise with property-based testing (with, eg randomized inputs across the domain). I’d be willing to guess it’s just a matter of education and/or desire keeping most developers from using it.
Personally I use python and solve real world problems.
The effectiveness of a Copilot-like tool trialed at FB showed that 8% of code contributed by participants was sourced from suggestions, but the latter study made no promise about coding velocity: https://arxiv.org/abs/2305.12050. In my own experience the time taken to review machine-generated suggestions often eats into developer time.
This is not a critique of LLMs in general — I’ve found ChatGPT really great for kicking off greenfield projects in well-known languages and frameworks.
Although I could be hallucinating the whole thing.
No idea if it feeds all of the code to the LLM or just parts, but it's pretty good at interpreting what the code does
Through it could do stuff like go through your repository(ies) and generate embedding for sections of it and then have a vector database + retrieval argumented generation (RAG) system.
Basically all the LLM needs to do is translate human writing to some format that a "normal" service can use. That can then leverage the existing spotlight system that's pretty decent at searching stuff on the phone anyway.
Then it'll report it back to the LLM which translates whatever format back to something humans can process.
Basically "less shit Siri"
The "chance of hallucinations" is the tricky bit - if I have to manually check everything it does in case it's hallucinating, then it's not actually a solution. It's not saving me time (as TFA says).
Especially those that are a bit tedious and almost mechanical, but not quite mechanical enough to do with a simple search-and-replace.
> which I don’t think VS Code can do as an automatic refactor using the native tools.
Yet another point for using an IDE over a texteditor with some bolted on IDE features, because the Jetbrains tools even suggest such refactorings ;)
It also makes it a lot easier for juniors to chuck random code at seniors for review.
The standardised bit seems more difficult than it should be as it's essentially a social problem with development in general, but that also is a problen AI does not address, it's merely a bandaid over the problem.
At first it was a massive pain because I didn't realise it wasn't a "proper" autocomplete(intellisense is probably the king in that regard), and get hit with a large number of hallucinated functions.
This was really hard for me, as I'm slightly dyslexic, which means spotting plausible but bullshit completions is very hard. (I suspect its hard for everyone else too). Worse still, at the time the linter/type inspector was/is very slow so only ran on save/execution.
However its both improved in the last year significantly, and I have got used to it. For me there are a few techniques that help me:
1) It has a recency bias. Which is great for when you're jumping about in code making changes
2) It rewards proper variable names
3) Your comments should say what, and why your doing what you are doing.
2 & 3 should be obvious and you should be doing it anyway. But it really re-enforces that.
However I would really like some UI changes so that I can make _better_ use of the LLM plugin.
1) a completely different colour to indicate that its an LLM suggestion (bonus points for giving a confidence as well)
2) a different keystroke to accept the suggestion. (bonus for partial selection)
I think the truth as revealed in your comment and many others is that it all depends on the context. For repetitive code or boilerplate, or starting new projects in well-known frameworks and languages, it probably does increase velocity. The other context factor is the programmer themselves. Their familiarity with the problem domain, the language, and the framework all matter, as does the personality and coding style of the individual.
For example, I use vanilla web components which have some boilerplate for new components.
I already have a simple vscode snippet that does the job well, with no hallucinations [1]. I've experimented with llms doing the same thing with not great results.
It took me longer to explain what I wanted than it did for me to just write that snippet. Doing it repeatedly and waiting for the results definitely didn't increase my speed, though I was impressed that it was eventually able to figure it out (vanilla web components with lit-html renderer isn't a super common technique). Also, I prefer the pythonic approach of snake-case local variable names. Getting the llm to do that in a js project where it's not super common was another whole iteration.
We've had code generation tools around for decades to deal with repetitive boilerplate tasks. Maybe they aren't quite as capable as when the llm "gets it right" - but I wonder with these productivity claims how they are measured.
Are they starting from zero and a new developer? Or comparing against an experienced developer with proficiency with lots of workflow enhancers like templates, snippets, vim macros, etc...
[1] https://gist.github.com/claytongulick/e4251c1b27b22c25fb68f3...
I'd rather slower coding speed, properly written as it provides higher overall velocity. And velocity should take into account the refactoring that happens months or even years later. Crappy code can look really fancy, and even be bug free, but if it's overengineered and hard to change, it can create long change times or even a full stop in development later in the products lifecycle.
And that's on top of developers losing the understanding of how something actually works. If AI helps create the code that would have been written without AI, then great, but I don't observe that happening, and the code has never been a better idea than then dev could have done without it.
AI will get there eventually, but this current paradigm seems increasingly only useful for spam and shitty clip art. Even so, everyone is throwing absurd amounts of investment capital at it in the hopes that something useful will happen. It's a pretty clear depiction of the investor class being so detached from the technical reality of what they're investing in that they just sit around lighting billions on fire thinking that they're getting richer instead of poorer. Go ahead and buy more H100s though...
Either I'm a dimwit, easily conned by hype and shiny tools to the point that I can imagine benefits for two years that simply aren't there... or there's something to them.
[1] https://simonwillison.net/2024/Mar/22/claude-and-chatgpt-cas...
I'm able to get really great results out of LLMs because I have 20+ years of experience helping me know what questions to ask of them.
I do feel like my rate of learning has increased though, because I'm much more likely to try out a completely new technology when I know an LLM can flatten the learning curve for me a bit.
It's the same for me. Lots of experience knowing what to ask for. It does a better job in summarizing knowledge and getting me a relatively coherent explanation. Much faster than using Google Search to find and connect the dots from dozens of pages.
I just don't see much benefit in its reasoning and code assistance features besides basic stuff.
That seems rather unlikely given that LLMs didn't exist more than seven years ago, much less 20+.
While a direct answer is nice, I like an iterative/explorative process because of all the things I pick up alongside it. An example is when I was working on an epub reader for macOS (side project). I wanted to a native layout engine instead of a webview and I decided to go with muPDF. This has lead me to know more about text layout and rendering, and embedding C inside Swift than I would if I just have direct answers for every problem (if I'd know the correct questions in the first place).
I accumulate side quests like that until I can do a nice experiment to learn as much as I can for a particular domain space.
I agree with the parent comment, my pipeline is already saturated with side quests, I'm already iterating on a bunch of random fun work, and side-project work and work work. So often times the most "LLM-heavy" projects of mine are things I straight up would not do if I didn't have something to get the ball rolling other than more of my own free time which is already in short supply.
Hopping from direct answer to direct answer isn't where I find wonder/fun in programming anyways, but sometimes you don't have bandwidth for the side quest to be fun or wonderous.
So I asked a couple LLMs. They wrote out loops for me to format the data how I wanted. I could have copy-and-pasted that in and it would probably have worked. But I felt there was something better yet, so back to Google I go.
It's `PDO::FETCH_KEY_PAIR`. It's built-in. But oddly kind of hard to find unless you know the right thing to search for, and "key pair" was not springing to my mind.
Point is, if you just let the LLMs do your work you won't even find these better ways of doing things. And I'm quite afraid of what the LLMs are going to do to Google, Stackoverflow and documentation in general. In a couple users it'll be ungoogleable too.
My gut feeling is that we're going to enter into a 'dark age' of coding where a lot of previously available resources are going to be ransacked and made hard to find in favor of big corporation owned LLMs. It's already having an extremely bad effect on search in general; we're potentially only a few fights away from sites like SO having users leave en masse. That's why I think having a strong network of engineers to talk with will become more important than ever, almost a return to the IRC days.
If you work in a small company and you are the most experienced developer, you don't often get feedback on how you can improve things.
The trick is, quite simply: just ask. I regularly dump some code I wrote in a language model and then ask what can be done better.
I would never do that in any online space, because first, I don't wait an answer maybe some day, I need an answer NOW. And second, I prefer to avoid being called a fool.
Too bad I'm "engaging with them wrong", I could have sworn it was helping me.
Seriously though, claiming LLM's don't have any higher level understanding of right and wrong and then extrapolating that to "they cannot possibly be used to improve things" is a very stubborn refusal of the fact that the most logical answer to the question "what can be improved here" is... actual improvements.
We tried pair-programming, and he cracked within a few minutes. He couldn't articulate his thought processes verbally. Sounds similar to the scenario you're describing.
You know there have been lots of other technologies people have been working on in the meantime and concurrently? Some of them more questionable, some of them less.
However I can’t help but notice that the vast majority of blogposts, talks, tweets, and basically everything else you do now is around LLMs. Do you not think that’s indicative of this being “hyped” and “shiny tools”?
Only if you're including things that don't exist. If you restrict yourself to things that people might conceivably talk about, that probability is actually very high, for the simple reason that bad things are exciting specifically because they are bad.
Here are some boring things:
- bread
- sunlight
- air
- water
- parents
Note that starvation, darkness, suffocation, dehydration, and being orphaned are all much more exciting than their opposites.
I don't see that as a hype thing - in the past I've had other topics I've focused on, not because of hype but because those were the topics I was spending the most time with.
What if we have already passed the inflection point where the exponential growth transitions to an s-curve? That would mean that this technology on its own would only get marginally better than is today. Maybe 2, 4 or even 10x better than it is today, but not 100x or 1000x. To break those barriers, we would need further innovations beyond just throwing more gpus and training data at the problem.
I personally am short on LLMs because I believe it is much more likely that we have already crossed the inflection point or will soon. Again, they are impressive but ultimately I think that LLMs will be at best a footnote in history if they are even remembered at all in a few hundred years. But of course I could be wrong.
My personal opinion there is that if all research froze today it would still take us years to figure out all of the potential use-cases and applications for the models we have access to right now, and how best to apply them.
I think it's always going to be a YMMV experience with LLMs. I'm an extreme generalist with 23 years of experience so it compliments my strengths and weaknesses. I know how to program across 20 odd languages but most of them I need to look stuff up if I'm not using it frequently enough. Now though, don't really need to look stuff up.
For me there are two groups, those that want to use LLMs and are pushing it forward and those that are would prefer that LLMs not exist and want it to be hype that goes away.
Having followed the hype cycles of VR/AR and crypto I can feel a difference. Both of those felt like a solution in search of a problem. The true believers wanting it to be something like Ready Player One with VR (fun story, Oculus/Facebook actually handed out copies of the novel at Oculus Connect 2 in 2015).
The hype around LLMs seems different; like applying a new solution to all the existing problems to see where it helps, and that's just with the current iteration.
[0]: I'd prefer to be using local/open equivalents but the capabilities are still lacking.
The first 100 lines of any project is spam (usually 0 entropy), so AI is a nice win.
Google is terrible in comparison, just let it write the code for you.
And it can feel like magic, because we're pattern-seeking and pattern-matching creatures, so something that seems to intuit the pattern we're looking for (even if imperfectly) can feel quite a bit like reasoning with another human. To be a bit less generous to humans, it's often the same as talking with another human, because we spend much of our lives writing boilerplate or making small talk or otherwise just kind of on autopilot and answering patterns with expected responses without seriously engaging our complex reasoning. (Do you have to really think about how to map-filter-reduce a dataset anymore? Or think seriously about how you really are when someone says "how are you"? We don't usually expect other people to, and most of the time we're satisfied by exchanging recognizable / somewhat coherent language patterns which we can count on each other to fill in the gaps of. But this is not the intelligence in human intelligence).
The corporate hype machine works overtime to hype it as a solution, but that's just what they do. That's all spin. We see through it now, and we're getting to the point where people are asking the only important question about this tech, which is, how much will you pay for a nerfed autocomplete to do X?
i think this is a super important for anyone playing in the LLM space to calculate.
currently “AI” providers are selling electricity at a loss to demonstrate product “value”, so even while asking questions is “free” today, there’s actually a finite resource under the hood that needs to have the bill paid in the end.
an estimate, but from historic trends, free bills come due in around 7 years
---
However I don't really think that 'if' will come to pass: I expect that we will still see lots of advances in the 'intelligence' of these models, so people will compete on these, and worry about electricity usage afterwards.
the CoPilot product has billed hundreds of millions of dollars, and is billing right now. Secondly they copied all the GPL code and put it into a mixer, like a bitcoin mixer in a legal way.
Sage words about "what they do" ring hollow while the only measure that counts is money. This situation needs legal action.
AI art is better than most humans, including most artists.
You can take the outputs and immediately turn them into animation. Suddenly I, as a single individual, can easily make film content without figuring out set and lighting logistics or roping in dozens of people.
This stuff is magic.
Would it win any awards? I’m pretty sure not but it’s more than adequate for its purpose and is almost certainly better than anything I could have come up with even if I used some CC artwork.
ADDED: I also had something of a vision for what I was looking for and (I think) enough of an eye to fiddle with things and get to the point where I went "This isn't half bad."
Conflating "art" (the "snobbish" definition if you will) with realistic depictions of imagined worlds, stories and settings is not helpful here. There are so many talented, imaginative people out there that have previously had Zero ability to manifest their thoughts into a visual form. AI has enabled that on so many levels and we should be grateful for it.
https://www.youtube.com/watch?v=G4wJ4WeJrz4
You also don't see the everyday people using it to generate pictures about their own experiences and lives, using it to populate their DnD worlds, their personal unpublishable fanfics, etc.
This part is important since I'm still yet to see evidence that the AI artist is capable of thinking more deeply than "(full_body:1.2) , best quality, (8k, RAW photo, best quality, masterpiece:1.2) , (realistic, photo-realistic:1.4) , ultra-detailed, (Kpop idol) , perfect detail, looking at the viewer, makeup, pretty South Korean lady wearing a bathing suit, wet skin, light reflections, angelic cute face, half body shot, short brownish black hair".
The linked video isn't there yet, but it seems obvious what would or wouldn't make it good, and the list of traits you describe is really all I want in an image generator. I don't care what the girl is thinking about; that doesn't show up in the image anyway.
I do want to be able to say "no, make the hair shorter", or "turn her to face more toward the left of the frame" or "have her pointing at the door in the background". If that command takes the form of a comma-delimited list of traits, so what?
Is it better than I could do? Yep.
Is it failing a bar that any human would pass? Yep.
Is it better than relevant artists? Nope. But it is cheaper.
This is the worst it will ever look. It's only going to improve from here.
Compare that to the early days of silent film. The profess with Gen AI is astounding.
We're going to have Disney/Pixar and Scorsese outputs by the end of the decade. (I'd be willing to wager even sooner than that.)
Did you mean to respond to my other, nearby comment? The issue I cite above isn't cherry-picked in any sense; it's a big, glaring problem with what was cited as an example of "good work". If someone's head is a balloon, the balloon should occupy the same position in 3D space as the head would.
> Compare that to the early days of silent film.
This is an interesting comparison. I don't think it really works. Early silent films were aware of what could and couldn't be done in the medium; there aren't any that rely on a nonexistent soundtrack. Generative AI stuff doesn't seem to be very concerned with "what kinds of things can we do well?". Instead, they're attempting everything, almost none of it is being done particularly well, and there are theoretical arguments over whether it makes sense to try to improve on individual tasks.
As I mentioned before, we are now capable, and are indeed creating, feature-length works.
Reality is: AI is startlingly expensive. Stupid, stupid expensive. Microsoft is building "Stargate", their $100B AI-focused supercomputer [2]. The money to build that isn't coming from selling AI services. Its coming from their old, boring, tried and true businesses; Windows, Office, Azure, and M365. Xbox is dying before our eyes. Their tried and true businesses make great money but aren't growing. Its up to AI to prove to investors that their P/E multiple is justified.
[1] https://twitter.com/tsarnick/status/1789107043825262706
[2] https://www.tomsguide.com/ai/meet-stargate-the-dollar100-bil...
But IMO the current AI hype has some almost religious elements to it
Any programmer that is this dismissive of these tools frankly... hasn't been using them right, and has no imagination - or a massive ego. This stuff is still far too early to make such judgements.
Statistical models will always be limited by their data and how edge case filled reality is.
I think, their ROI model is much different and subtle. Few decades back, everything ran on-premises and costed ok ... Then a huge investment had been shovelled into fancy frameworks and pushing strange paradigms (microservices, distributed systems, monoliths whatever flavour works for the tech giants) so now people just put together whatever and needs massive _cloud_ to run basic things that used to run on few MHz devices.
In the end, the ROI is NOT from these models, those are just gambles in case one actually becomes a king maker. The actual ROI is from the _cloud_ where innocent people will rent expensive hardware to try and utilise these models and funneling wealth to the actual investments(cloud operators, hardware vendors) which are already part of portfolio of these investors. Basically, $1MM invested in random shiny startup will inspire other shiny startups to also spend similar in race to become unicorn, all the while the cloud operators and GPU vendors are laughing to the bank preparing their shareholders(usually same investors) dividends.
I don't get that feeling from the LLMs. They have about the same skill level of an intern, but they don't _learn_. I can't offload any work to them, and they take the same level of effort to manage.
I'm not in this job to manage interns. I'm in this job to solve problems. Training the intern is a payment by current me for future me.
This isn't like going from books to Internet search or StackOverflow. It doesn't provide an immediate benefit to me, and I don't benefit in the future from my contributions. I'm not seeing the share-alike vibe necessary for scale.
I want tools that make me faster and more efficient. That's how I keep increasing my wage. Maybe if I could _see_ the AI learn from my training? Maybe if I saw benefit from the effort? However, right now, I'm paying for the benefit of training the LLM.
The problem is that there’s always a communication cost to outsourcing something. And you can’t outsource a whole project to an LLM, like a human intern. You’re just outsourcing one micro-, sub-task after another. To use the human analogy, it’s more like you’re standing over their shoulder and telling them what function to write, one after the other. And they’re SUPER fast with small, pure functions, but they get confused with anything else.
Is that a faster way to program?
Maybe? If you can fluidly decompose things into small functions in your head and the problem can be solved that way?
I don’t know though, I find myself using chatGPT for bigger meta questions much more frequently than the Copilot autocomplete
https://github.com/community/community/discussions/58059#dis...
GPT-4 Turbo - the much cheaper version of 4 - was announced on November 6th. So there's a good change they've switched to that at some point in the last 6 months.
If it doesn't monumentally increase the speed, it helps devs to stay productive longer
When I review code, I am completely unable to get this mental picture, and I usually miss a lot of issues that I would otherwise have avoided.
Which is why I don't see the point of Copilot.
I agree 100% that reviewing human code is a lot of work but not with copilot IMO because of the immediacy of it all.
There is no possible way to 1. get a complete picture 2. understand all the constraints
because we pull data from the real world, and the real world is constantly changing, and we still don't even have a single theory of everything in physics.
Another time I asked to generate a code to listen to SMTP server using a library, and again it's easier to review because I was not familiar with the library. From that I can make my own adjustment.
How does copilot et al. speed up verification and review? For example, a C program is harder to verify than a Go program doing the same thing.
Suppose you use it to verify something in an enterprise context. Are you confident in the result? Personally I'm too paranoid for that and I think more people should be
Which is a very dangerous place to be in, because the more you assume that the code generated is reliable the more likely hallucinations are to sneak into your codebase. And now you don't have a good mental model of what your code is doing because you didn't actually write any of it.
I have a feeling that codebases using and relying on copilot are, essentially, pushing disasters later down the line when bugs pop up and no one has the ability to actually debug the system.
and then you look at the resulting flatmap, filter, reduce blob of AI generated code and figure out if what it does is correct for about a minute.
I think if you regularly use CoPilot to write entire functions or use its prompt mode, you will spend more time verifying its output is accurate than it would save writing the code manually. If instead, you use it iteratively via autocomplete to write small fragments of code, a line or two at a time, its trivial to verify its correctness, and it will save you a good 20-30 seconds at a time, adding up to large savings over time.
I exclusively do the latter, so I find it incredibly useful. The few times I've tried using the prompt mode, or comment-driven code generation, it's been very average or awful.
My guess is the people that don't use Copilot because they don't trust it, or don't like the code it produces, are probably trying to use it in a professional scenario where they have strict coding standards. For hobbyist use or side-projects, Copilot is an incredible time saver. I use it with my Python side-projects, and it's truly amazing how much time it saves me. Some of the most common use-cases where it saves me time:
- Adding docstrings. Just type `"""` after a method name, pause for a second, and Copilot will generate a pretty decent docstring. And usually it will mimic the style you've used to write other docstrings in the file. - Writing tests. This one can be a bit hit-and-miss, but I mostly get good a good starting point, and in the best case it will suggest test scenarios that I might not have thought about. - Creating basic functions/methods. I have terrible memory and usually need to refer to the docs for even fundamental Python code, e.g. reading or writing CSV files, looping over and modifying dictionaries, etc... But with Copilot, I get great results, often just by typing `def open_csv_file` and then pausing for a second to get the method code. If that doesn't work, adding a comment with what you want the function to do before writing the function name will help a lot. - Adding type hints to existing code. Just type ":" and pause for a second, and Copilot will almost always provide the correct type hint. And pressing a space after the closing parentheses in a function, will almost always return the correct return type hint.
I just can't justify paying $19/month personally for the "Business" plan so they don't use my proprietary code to train their AI.
Why not? Is it a significant fraction of your income? I find that the cost of hardware deprecation, electricity and internet access probably amounts to a lot more than that per month for me already. (Not to mention the opportunity cost of my time.)
It's fine to be frugal, but it's also useful to compare your actual costs.
(Though that's not always appreciated. I got into some minor trouble at an employer when I pointed out that the 15 GiB of storage space they 'graciously' gave us should cost them less than the company toilet paper we use every day. And I said 'should' because that's internal storage space, and I used numbers from AWS. But if they are paying significantly more for storage than what AWS charges (with enormous margins for Amazon), we also have a big problem.)
Knowing what to write and how to organize the code is the really difficult part of our job, and I am unable to do this correctly when reviewing stuff written by others (or by an AI).
I have been wondering if people are actually using those to save time writing boilerplate code, in which case it would make sense.
Vim has greatly increased my productivity by allowing me to save time for all repetitive tasks, and it's consistent and reliable (but also super hard to learn).
Maybe it's just a completely different approach in resolving the same problems?
And I like to have an idea of everything I have to manipulate, variables and property, functions and classes, modules and files. If something has not been written by me, I have some good reasons to trust it, either because I reviewed its source code or the library reputation (I'd probably read the source code anyway because some documentations are not good). IDEs have been able to provide the kind of completion that I like (auto-importation, symbols) and information I want (function signature and class properties).
I'd rather take a moment to read a library's documentation and code examples than see ChatGPT generates something that has a non-zero chance of being an hallucination.
This is the ideal place to add the most value, since a generic LLM will be weakest here.
A couple of days ago I was building up a Dockerfile for a new Rails project and I literally just let Copilot build it for me instead of look it up. It was probably 95% correct.
> It definitely hits a wall when I am working on something sufficiently esoteric.
So do humans. Hiring humans to do that work well is hard and expensive, so it’s not strange that is harder for LLMs as well.
I am not sure that finding things LLMs are shite at makes them less productive for most things programmers are doing during the day, which is quite boring stuff. In my reality, which, before mid last year, had 100s of people (across clients I work with, not my company) doing integrations and frontend work, now fired most of these and focused more on optimising the pipeline for copilot, chatgpt and opus.
My friend who runs an outsourcing operation in India almost went out of business mid last year when he saw his EU clients cancelling all his contracts because his people could be replaced by openai; he pivoted by having his people use ai, do ai projects and becoming more (everyone already was that but it was more heavy on the coding) business analyst heavy. He fired no one and is doing better than before now, but he doesn’t hire ‘programmers’ anymore.
My company does the same but in a niche; our specific tooling is easier to automate (which we did/are still doing) and we notice it is getting easier with the llms getting better. A year ago we still needed a human to evaluate even the basic scope of work, now llama3 + some tooling we built can automatically get us a locally generated report.
But this is what I want to know. They thought they could replace the ppl in the first place.
If Microsoft focuses copilot on simply making things more enjoyable rather than do the work for you, I think it will be an amazing thing.
I also think that co-pilot vs code integration extension is big part of it. I don't think I would be as productive with chat UI alone. I wonder if there is alternative I can use with local hosted llm that can give me full block autocompletions similar to co-pilot extension. Everything I've seen so far was at best line at a time autocompletion.
I'm not sure what you're working on, but as someone dependent on LLVM-based tools, reading this is kind of terrifying. I'm sure your intentions are good, but I hope you're at least making it clear that you're heavily reliant on the output of Copilot when making contributions to something. C and C++ in particular are not the kind of language where someone inexperienced can be trusted to review.
> Yes, the code it generates come out often incorrect, but even my low level expertise allows me to spot issues.
Can you give an example of some issues? C and C++ are compiled, so there will be obvious compilation errors, but there are also a lot of "foot guns" that are not obvious and will bite you at runtime.
The thing is - I don't think I would dare to do that I've done without co-pilot in the first place. So it is a win anyway
And given that I've worked in the languages I write for a living, mostly C and Python for years it's not like it ever suggested anything I wouldn't have known how to write and especially in C it can produce some catastrophically shoddy code.
The one case where it's kind of nice is if you do a side project with a library that doesn't have good documentation and you don't know the API and it spits out some nice example, but honestly I don't consider that worth paying for.
This conversion code was in service to using Plotly (the JS graphing library) for the first time. I was trying to get the axes of my surface plot to be labelled with the correct values (instead of just their index), and spent probably a half hour looking at the demos and the docs, and not finding what I needed. I asked CoPilot in the VSCode chat window about it, and it gave me exactly what I hadn't been able to find. As soon as I saw it, I understood the natural extension of the API call I was seeing in the demo, but I still hadn't found a second data point to even start extrapolating, if you take my meaning.
Anyway, I'm paying the $10/mo out of my own pocket, and yesterday -- at least to me -- it paid for itself for the month in just those two examples. I find it delightfully surprising. I just need to find a way to make it stop suggesting comments. I can type my own comments, thank you very much.
Autocomplete for files & paths worked great before copilot got involved.
There are occasions where I'm just not in the mood to work something complex out so sticking a temporary comment in telling it what to do can sometimes yield a good enough result, all be it with some occasional minor tweaking.
Where it's really shined is with repetitive tasks. It's nothing close to perfect but it's certainly a welcomed little helper even if it does get things wrong sometimes.
In terms of the original question, I think its the cognative mental load thats being helped here more than anything, not having to waste brain power on the mundane bits is the big selling point for me.
Sometimes, it was magical, but more often it was mediocre and just got in the way.
I generally have better luck with a back-and-forth chat with an LLM on the web and then cutting/pasting the snippets into my IDE. I can do those for free.
Will try again in a year, probably.
I find copilot mostly to be a useful Google replacement, e.g. if I forget the syntax for a for loop in JavaScript I can just write a comment like `// For loop over X` and it will spit out the boilerplate I need without breaking my flow.
In either case it's really important not to get stuck "trying to make it do what I want". If an LLM can't solve your problem in 3 tries, you're probably asking too much of it.
Copilot helps me a lot when I have to write repetitive code, or when I have to change code slightly the same way in a lot of lines.
I never add large amounts of code at one go though, I think it would certainly add subtle bugs then.
I'm still not sure if this is a transformative way to code or not. I think in the future it might be able to do to programming languages what programming languages did to assembly... but it probably won't look like a chat UI or an autocomplete plugin.
> You might be surprised to learn that I actually think LLMs have the potential to be not only fun but genuinely useful. “Show me some bullshit that would be typical in this context” can be a genuinely helpful question to have answered, in code and in natural language — for brainstorming, for seeing common conventions in an unfamiliar context, for having something crappy to react to.
> Alas, that does not remotely resemble how people are pitching this technology.
It seems to me a good IDE generates most of the boilerplate code I would need to write. For Java I use the community edition of IDEA, that's open source, it runs entirely locally and there's no lag. No open source code gets eaten and no forest get burnt.
I spend a vanishingly small time writing boilerplate code. Or so I think. Am I just not noticing it, or is it rare and I work with high quality code?
For me it felt like having someone trying to finish every sentence I’m speaking: even if they’re right 100% of the time, it’s still very distracting and more than a little irritating.
And I wonder if different types of people react to it as differently as the stutter/non-stutter dichotomy.
And I wonder if different types of people react to it as differently as the stutter/non-stutter dichotomy.
Maybe I'm dumb but these reasons aren't obvious to me. Why doesn't the author use copilot at his day job?
If you want to make this argument, then the realistic reason would be that Copilot hasn't yet completed the SOC 2 compliance process. That would be a valid reason to wait, for corporations that are listed on Nasdaq or work with very sensitive data. But that's far off from the comment I'm replying to.
Give it existing code to work with, particularly a larger code base and things get weird.
Still, I love it for hacking out small prototypes. If you don't really care about performance or best practice, it's ok.
By larger I'm talking about maybe 5 or 6 files.
Or to be honest Chat GPT isn't good when your fixing code that already exists.
I see this claim a lot, and I have no idea whether or not it's true. I just struggle to imagine that there are millions of people employed, well compensated for doing pretty easy jobs.
In my professional work and my various unrelated personal projects and open source contributions, if you want to do anything useful or interesting, you get into the weeds very very quickly. The territory where LLMs are nearly useless. I'd say maybe 5% of my time is spent writing "easy" code without major complexities.
Overwhelmingly in my experience of reading these back-and-forths, they are referring to 3.5 at best (as in this article).
Yes, of course I accept there are users of the paid version (4+) and class competitors like Opus that don't find they help their productivity but unfortunately any comments referring to significantly less capable engines than those - without at least clarifying that - risk just adding noise to the conversation.
It’s like a phone number. If you never dial your mom anymore, you will forget it.
Does the author mean that Copilot filled in the other `switch` cases? (Because that looks like "Here, let me plagiarize that for you. We'll call it 'AI' and 'boilerplate'.")
What an interesting, nuanced take. Thanks for sharing, you've given me a lot to think about.