Chris_Newton

A simple task, migrating a typical password reset flow as part of updating a long-lived web application from legacy libraries and software architecture to modern equivalents, apparently cost roughly the same as 2 months of Pro subscription, over the equivalent of about half a working day in wall time.

It produced code of decent quality at a small scale, but it wasn’t always on point architecturally. It also had a tendency to drift off topic and try to tangle up other changes it decided should be made with the main change we were supposed to be working towards. So even for a routine task, based on a plan developed using the harness first and with the agents working under close supervision, a near-SOTA model is still producing quality on par with a decent mid-level developer but substandard for anyone senior+ in this case.

Moreover, based on a direct comparison with other migration tasks of similar complexity that I’d already done by hand, it was actually a bit slower overall to work this way. I had to babysit Claude throughout and review everything it proposed carefully, both to avoid subtle errors (it would have made several) and to prevent drifting off track. I also had to spend a significant amount of time cleaning up its final output to an acceptable standard after the session. Those two overheads more than cancelled out the much faster code generation an LLM offers under favourable conditions.

So for now, I remain sceptical about these high multiples of improved productivity that I keep seeing claimed online from people who are apparently writing almost everything using AIs now. I could certainly have achieved a multiple of my normal productivity by YOLOing everything without reviewing it in detail and then accepting the output code without tidying anything up. However, I doubt this codebase would still have been good enough for normal human developers to work on it reasonably after even 10 or 20 AI-led sessions like that. The architecture would have degraded significantly and the test suite would have been large and largely pointless. And again, this wasn’t rocket science in this experiment, it was completely unremarkable maintenance of a relatively small and simple web application.

Hadn’t good developers been focussing on those things, and other high-level modelling that relates the software system back to the underlying real world domain, for many years before the LLMs were in the picture?

I’d even say that it’s one of the most reliable markers of a more senior/experienced developer that their code reads like a clear and logical explanation of what the system does and why, with concerns well separated and minor details and technicalities abstracted away so they don’t clutter the rest of the code.

I agree with the spirit of what you wrote, but my recollection of the timeline is different. The first decade of the 2000s was peak Crazy Agile Advocacy, but IIRC it wasn’t until the 2010s that unit testing really became almost universal practice. Much before that and it was still tangled up with XP, TDD and lots of other things that certainly weren’t universally accepted as good practices (notwithstanding the strident advocacy of a certain group of consultants/authors/speakers/bloggers and their fans).

I remember, back in the mid-2000s, when we had some consultants brought in to talk about different aspects of quality and testing. There were several working groups, each led by one of those external consultants, and one of them was about unit testing. This was in a relatively large software development organisation for the time, a few thousand people, and while some parts of the organisation had some form of automated testing operating by then, it definitely was not the case that the well-known products produced by the organisation all had a unit test suite. Other practices we’d consider routine today, such as peer code reviews, were also in their infancy during that period: some were doing them, many were not, and generally we had much less experience of how to do them effectively than we have today.

As an industry, I don’t think we really matured in how even the most ardent fans of unit testing were writing test suites until the 2010s either. In the 2000s, we still had lots of people mocking the entire universe and then writing unit tests that were 99% testing those mocks because of 100% test coverage requirements, and similar dogmatic nonsense.

By the 2020s, I think there was much more awareness of that automated testing is generally a good idea, but there are different kinds/levels of automated testing and finding a mix that suits each project’s specific needs is important. One of the great benefits from the more recent AI tools, particularly the agentic ones over the past year or so, has been that it has clearly demonstrated both the value of a good automated test strategy and how much of a waste of time vacuous tests are.

Meta-moderation was an interesting idea, though they seemed to have stopped promoting it much by the time I stopped posting there often. I’m not convinced it’s as effective as having a small pool of “super-moderators” who can not only affect the prioritisation/visibility but also comment themselves to guide contributors in positive directions, but practically speaking, it might be more scalable.

On the positive side, all of the above have attracted many people to their communities who have contributed useful or interesting points. We all give away our thoughts and experience for free while participating in these discussions, but we gain in return from the freely shared knowledge and experiences of others. I also appreciate those who take the time to vote/moderate so that the best contributions stand out. Overall I find these online discussions extremely valuable and I’m sure others do as well.

On the negative side, there are some common failure modes. There have always been the trolls who will post offensive or misleading comments, and even when it’s a small minority, they can be disproportionately disruptive. There have always been the Dunning-Kruger contributors who would insist they were correct even as others tried to explain why they weren’t, and then the people who do know what they’re doing feel obliged to waste time repeatedly setting the record straight so no-one comes along later and gets misled by the incorrect or misleading contributions. I will never understand the current fascination with getting AI bots to contribute mediocre or just plain wrong comments in these discussions. But the worst recurring pathology by far, IMHO, is when there is some form of community moderation but that goes off the rails. It killed SO by deterring good contributors for petty reasons. It has killed many a promising subreddit; I have recently given up participating in several myself that used to be interesting, because their moderators started killing entire posts retrospectively, which repeatedly cut off discussions where some contributors had already taken the time to write up good solutions to someone’s problem or share their relevant experiences.

I’m not sure anyone has really got this right at scale yet. On smaller sites like HN, the moderation can be very good, but that relies on the fact that it can be managed by a small number of decent people. If your community is big enough that it needs to be more self-policing then the time-honoured question of quis custodiet ipsos custodes? is as relevant as ever. I strongly suspect that the only real answer to this is some kind of hierarchy where the operators of a forum set culture from the top, then just as a few negative contributors can spoil things for everyone and so some form of moderation is introduced, so a few negative moderators can spoil things for everyone and so some ability to guide or if necessary remove the use of moderation privileges is needed.

Personally, I’m content being a dinosaur who writes one letter at a time (in handwriting that has been praised for its neatness and clarity ever since I was at school myself) or uses computers to render the text for me (where I have long had an interest in typography and quite enjoy making pretty text using elaborate cursive fonts, but for special effects and interest, not for body text and legibility).

For reasons unknown, LinkedIn seems to have decided that I’m not me a few months ago and blocked my account, though it would apparently be willing to reconsider as long as I provide whatever it is that Persona wants these days. (Evidently contacting me directly via my company — where my role as one of the directors is a matter of public record and my email address was listed in my LinkedIn profile — was too much trouble. :sigh:) Since I have no interest in giving any personal information to Persona, I no longer use LinkedIn and remain blissfully ignorant of all the AI-driven content that I keep seeing complaints about, but I do miss the occasional good news stories about people I actually know. I should probably send a formal GDPR request at some point, since my profile is probably quite misleading by now.

Sorry, but I don’t understand what you mean here. What do we win by being faster at producing the wrong things?

I’ve been noticing that this simple reality explains almost all of both the good and the bad that I hear about LLM-based coding tools. Using AI for research or to spin up a quick demo or prototype is using it to help plot a course. A lot of the multi-stage agentic workflows also come down to creating guard rails before doing the main implementation so the AI can’t get too far off track. Most of the success stories I hear seem to be in these areas so far. Meanwhile, probably the most common criticism I see is that an AI that is simply given a prompt to implement some new feature or bug fix for an existing system often misunderstands or makes bad assumptions and ends up repeatedly running into dead ends. It moves fast but without knowing which direction to move in.

I completely agree about the problem of someone deciding to turn these kinds of scanning tools on and then expecting they’ll Just Work. I do think the better tools can provide a lot of value, but they still involve trade-offs and no tool will get everything 100% right, so there will always be a need to review their output and make intelligent decisions about how to use it. Scanning tools that don’t provide a way to persistently mark a certain result as incorrect or to collect multiple instances of the same issue together tend to be particularly painful to work with.

Clearly it is still possible to generate a false positive if, for example, CodeQL’s algorithm thinks it has found a path through the code where unsanitised user data can be used dangerously, but in fact there was a sanitisation step along the way that it didn’t recognise. This is the kind of situation where the theoretical result about not being able to determine whether a semantic property holds in all cases is felt in practical terms.

It still seems much less likely that an algorithm that needs to produce a specific demonstration of the problem it claims to have found will result in a false positive than the kind of naïve algorithms we were discussing before that are based on a generic look-up table of software+version=vulnerability without any attempt to determine whether there is actually a path to exploit that vulnerability in the real code.

I too declined on privacy grounds.

I’d be interested in what kinds of false positives you’ve seen it produce. The functionality in CodeQL that I have found useful tends to accompany each reported vulnerability with a specific code path that demonstrates how the vulnerability arises. While we might still decide there is no risk in practice for other reasons, I don’t recall ever seeing it make a claim like this that was incorrect from a technical perspective. Maybe some of the other types of checks it performs are more susceptible to false positives and I just happen not to have run into those so much in the projects I’ve worked on.

One security checking tool that has genuinely impressed me recently is CodeQL. If you’re using GitHub, you can run this as part of GitHub Advanced Security.

Unlike those naïve tools, CodeQL seems to perform a real tracing analysis through the code, so its report doesn’t just say you have user-provided data being used dangerously, it shows you a complete, step-by-step path through the code that connects the input to the dangerous usage. This provides useful, actionable information to assess and fix real vulnerabilities, and it is inherently resistant to false positives.

Presumably there is still a possibility of false negatives with this approach, particularly with more dynamic languages like Python where you could surely write code that is obfuscated enough to avoid detection by the tracing analysis. However, most of us don’t intentionally do that, and it’s still useful to find the rest of the issues even if the results aren’t perfect and 100% complete.

Yes, but the “MVC” pattern used by various back-end web frameworks that borrowed the term a while back actually has very little to do with the original MVC of the Reenskaug era.

The original concept of MVC is based on a triangle of three modules with quite specific responsibilities and relationships. The closest equivalent on the back-end of a web application might be having a data model persisted via a database or similar, and then a web server providing a set of HTTP GET endpoints allowing queries of that model state (perhaps including some sort of WebSocket or Server-Sent Event provision to observe any changes) and a separate set of HTTP POST/PUT/PATCH endpoints allowing updates of the model state. Then on the back end, your “view” code handles any query requests, including monitoring the model state for changes and notifying any observers via WS/SSE, while your “controller” code handles any mutation requests. And then on the front end, you render your page content based on the back-end view endpoints, subscribe for notifications of changes that cause you to update your rendering, and any user interactions get sent to the back-end controller endpoints.

In practice, I don’t recall ever seeing an “MVC” back-end framework used anything like that. Instead, they typically have a “controller” in front of the “model” and have it manage all incoming HTTP requests, with “view” referring to the front-end code. This is fundamentally a tiered, linear relationship and it allocates responsibilities quite differently to the original, triangular MVC.

The model can be completely unaware of any specific views or controllers. It only needs to provide an interface allowing views to observe the current state and controllers to update that state.

In practice, views and controllers usually aren’t independent and instead come as a pair. This is because most modern UIs use some kind of event-driven architecture where user interactions are indicated by events from some component rendered by the view that the controller then handles.

My go-to example to understand why this architecture is helpful is a UI that features a table showing some names and a count for each, alongside a chart visualising that data graphically. Here you would have a model that stores the names and counts as pure data, and you would have two view+controller pairs, one managing the table and one the chart. Each view observes the model and renders an updated table or chart when the model state changes. Each controller responds to user interactions that perhaps edit a name or change its count — whether by typing a new value as text in an editable table cell or by dragging somewhere relevant in the chart — by telling the model to update its state to match (which in turn causes all views observing the model to refresh, without any further action from whichever controller happened to be handling that user interaction).

In practical terms for a React application, we might implement this with a simple object/Map somewhere that holds the names and values (our “model”) and two top-level React components that each get rendered once into some appropriate container within the page. Each component would have props to pass in (a) the current state and (b) any functions to be called when the user makes a change. Then you just write some simple glue logic in plain old JavaScript/TypeScript that handles keeping track of observers of the model, registering an observer for each top-level component that causes it rerender when the state changes, and providing a handler for each type of change the user is allowed to make that updates the state and then notifies the observers.

There are lots of variations on this theme, for example once you start needing more complicated business logic to interpret a user interaction and decide what state change is required or you need to synchronise your front-end model state with some remote service. However, you can scale a very long way with the basic principle that you hold your application state as pure data in a model that doesn’t know anything about any specific user interface or remote service and instead provides an interface for any other modules in the system to observe and/or update that state.

The common criticisms are largely true: it does start slow with some weak episodes in season 1, some of the acting is a bit wooden, the CGI hasn’t aged well, season 5 is slightly anti-climactic because they largely wrapped up the main plot arc in season 4 in case the final season didn’t happen.

On the other hand, it had an epic storyline that spanned not just episodes but multiple seasons in a way that no-one had really tried in sci-fi before. That storyline made sense and weaves in and out of the individual episodes because it was planned out in advance. The world-building and development of different cultures and how they relate is generally strong.

Against that over-arching backdrop, it also had a lot of good individual episodes. They had genuine character development. They explored social and moral issues as well as any show of that period. They varied from diplomatic and political settings to the adventure of deep space exploration to almost pure action episodes. They varied in scale too, from relatable stories about a single individual, to stories about a whole planet or culture, right up to the fate of the known universe.

Much of the acting criticism is directed at the main leader characters, but I’ve always thought this is slightly unfair, because the script often relies on those characters to carry the plot and provide much of the exposition and those tend to be the more formulaic parts. The same show also features some of the best acting and main character arcs in TV sci-fi, with the relationship between Londo Mollari (played by Peter Jurasik) and G'Kar (Andreas Katsulas) being one of the great double acts. There were many good moments from the rest of the ensemble cast too, from the doctor wrestling with his conscience to a certain wave. And then there were some great supporting/recurring roles, from the light relief of Zathras (and Zathras, Zathras, Zathras, Zathras and Zathras, of course) to the much more serious Bester (arguably Walter Koenig’s finest work).

If you haven’t watched B5 and you’re a fan of epic space sci-fi, I highly recommend it even with its flaws. The first season is a slow burner (although it also has a lot of subtle set-up that you won’t appreciate until much later) but it picks up. If you’re the type of viewer who can’t stand filler episodes, there used to be some relatively spoiler-free guides to which early episodes you really need to watch and which you can skip, so you could look for one of those. Don’t watch In The Beginning first, though; it’s a prequel TV movie that has lots of spoilers about the main story that you’re not supposed to know yet when you watch the early series.

Windows appears to be positioned more as a platform to reach all the online services now, rather than its traditional role as a desktop OS. Can you even activate it without being online and having a Microsoft account any more? I’m out of the loop, so genuinely don’t know the answer to this one.

Office — or whatever it’s being called after the recent changes — also appears to have morphed into something quite different. I tried searching just now to see if you could still buy a permanent licence and install the classic applications like Word and Excel locally, and some sources implied you could, but I didn’t actually find any way to buy it in five minutes of looking around office.microsoft.com. As far as I saw, that site is now 100% about the online SaaS version and trying to get users to save their documents in the cloud. For businesses, the strategy seems to include promoting other online services like SharePoint and Teams as well.

So I think I stand by my original argument, though I don’t think it necessarily disagrees with yours. Windows and The Software Product/Service Formerly Known As Office might still be a significant part of Microsoft’s sales funnel, but they aren’t the products that Windows and Office used to be any more. The products they used to be have been repurposed to support an online-first corporate strategy, along with almost everything else in the Nadella era. Would Microsoft care if 100% of their customers stopped using Windows tomorrow and jumped to Apple or Linux systems, as long as they still used the other services that generate most of Microsoft’s revenues these days? I’m not entirely sure they would.

Again, is it really, though? I have no special insider knowledge so perhaps this is just a misunderstanding of the public information, but just going by the organisation structure, leadership comments and recent financials, it looks like Windows makes up a relatively small part of Microsoft’s revenues these days, while the traditional desktop Office applications seem to be almost lost in the noise. The emphasis seems to be firmly on cloud services, though admittedly with all the rebranding from Microsoft lately, I find it hard to understand even what basic products and services they offer any more.

Has it really, though? Or has it just shifted its corporate priorities away from its traditional stalwarts of Windows and Office, but in doing so caused disruption to users that had bet on the eternal stability of Microsoft’s product line? I don’t like the current direction of Windows any more than the next guy, and personally I’ve made other choices in recent years, but as a general principle, I’m not sure how reasonable it is to expect a business to continue offering the same product or service indefinitely if market forces are pushing it elsewhere.

IMHO, a deeper problem here is that we collectively allowed a near-monopoly culture to develop around desktop operating systems and basic business software. Instead of having a healthy degree of competition between providers and using standardisation to ensure interoperability and portability of our data, we’ve ended up in a “too big to fail” situation where many users have all their eggs in one basket and that basket has a rapidly growing hole in the bottom and looks like it’s going to fail anyway.

There are also reasonable arguments to be made about length of support for products already sold, forced obsolescence and ratcheting “upgrades”, where possibly the actions of some providers in the market are exploitative in ways we should not allow, and therefore regulating to prevent the undesirable behaviours might be in the public interest.

Ultimately, I think a combination of restricting customer-hostile practices while also encouraging a healthy degree of competition and interoperability in important markets would be best for the users and fair to the developers. Sadly, right now, we have neither of those things, and that’s how we get Windows 11, the mobile device duopoly, numerous examples of products or services being locked down against their users’ interests, online services that people increasingly rely on for fundamental aspects of their normal lives and yet that have little real obligation to those people in return, and assorted other ills of the 21st century tech landscape.

Not so long ago there was some sort of problem with Hydra builds for a recent version of Node. That seemed to result in trying to build the whole thing locally on every update, taking a huge amount of time and then typically failing there as well.

I’ve seen things with Nvidia drivers vs Linux kernel versions as well. We did have a specific reason for choosing Nvidia for that particular workstation, but otherwise, I’d agree with popular advice to get AMD if you’re building a Linux box, just based on the frequency and severity of Nvidia driver issues we’ve seen here.

I’ve seen a few issues with Ubuntu upgrades over the years as well, and wouldn’t necessarily rate that much higher for stability. That’s always surprised me because IME Debian Stable is the gold standard — something I’ve trusted with our production servers for well over a decade now, from unattended upgrades to several major new releases, and barely seen a flicker of a hint of anything breaking in all that time. To be fair, I haven’t used Debian much on workstations, so I don’t know whether the kinds of issues I’ve experienced with NixOS and Ubuntu would have been more common if I had.

Contrary to the name, even the stable branch of NixOS can have problems while installing routine updates with `nixos-rebuild switch --upgrade`. In fairness, at least with NixOS you can normally roll back to a previous working configuration where you can try to fix or work around the problem if that does happen. It’s still painful if you have to do that, though.

Even if your routine updates all go smoothly, as you mentioned, each stable release is only supported for a very limited time window after the next one is out. NixOS doesn’t have any long-term support branch in the sense that some distros do. Again, you can overcome this to a degree by customising your configuration if you need specific versions of certain packages, but in doing so you’re moving back towards manually setting things up and resolving your own compatibility issues rather than having a distro with compatible packages you can install in whatever combination you want, which reduces the value of using a distro with a package repository in the first place.

To be clear, I’m a big fan of NixOS. I run it as my daily driver on a workstation where I do a lot of work on different projects for different clients. Its ability to have a clean, declarative description of what’s currently installed globally or for any given user or even when working in any given project directory for any given user is extremely valuable to me.

But it’s also fair to say that NixOS is not for everyone. It has been by far the least stable Linux distro I have ever used, in the sense of “If I turn my computer on and install the latest updates from the stable branch, will my computer still work afterwards?”. If you’re looking for a distro you can deploy and then maintain with little more than semi-automatic routine updates for a period of years then, at least for now, it is not the distro for you.

I was writing code in an unrelated programming language at the time, and the bizarre inclusion of that particular file in the output was presumably because the name of the library was very similar to a keyword I was using in my existing code, but this experience did not fill me with confidence about the abilities of contemporary AI. ;-)

However, it did clearly demonstrate that LLMs with billions or even trillions of parameters certainly can embed enough information to reproduce some of the material they were trained on verbatim or very close to it.

IMHO, the trouble with that stance is that it leaves no path to incrementally update a long-lived system to benefit from any of those improvements.

Suppose we have an application that runs on 2025’s most popular platform and in ten years we’re porting it to whatever new platform is popular in 2035. Personally, I’d like to know that all the business logic and database queries and UI structure and whatever else we wrote that was working before will still be working on the new platform, to whatever extent that makes sense. I’d like to make only some reasonably necessary set of changes for things that are actually different between the two platforms.

If we can’t do that, our only other option is a big rewrite. That is how you get a Python 2 to Python 3 situation. And that, in turn, is how you get a lot of systems stuck on the older version for years, despite all the advantages any later versions might offer.

At the top, you have the true standard library for the language. This has very strong stability guarantees. Its purpose is twofold: to provide universal implementations of essentials and to define standard/baseline interfaces for common needs like abstract data types, relational databases, networking and filesystems to encourage compatibility and portability.

Next, you have a tier of recognised but not yet fully standardised libraries. These might be contributed by third parties, but they have requirements for identifying maintainers, appropriate licensing and mandatory peer review of all contributions. They have a clear versioning policy and can make breaking changes in new major releases, but they also provide some stability guarantees along the lines of semver and older releases are normally available indefinitely. The purpose of this tier is to provide a wider range of functionality and/or alternative implementations, but in a relatively stable way and implementing standard interfaces where applicable to improve portability.

Finally, you have the free-for-all, anyone-can-contribute tier. This should still have a sane security model where people can’t just upload malware scripts that run automatically just because someone installed a package. However, it comes with few guarantees about stability or compatibility, except that releases of published packages will be available indefinitely unless there’s a very good reason to pull them where you obviously wouldn’t want to use one anyway. A package you like might be written by a single contributor who no longer maintains it, but if someone does write something useful that simply doesn’t need any further maintenance once it’s finished and does its job, there is still a place to share it.

Firstly, you probably have a variety of analytic shapes to represent — things like lines and circles in 2D or cubes and spheres in 3D. Even seemingly simple questions, like whether two such shapes intersect or not, can require a significant amount of logic to calculate the answer. That logic will often be specific to the exact combination of shapes you have, because the number of freedoms and nature of any symmetries in the shapes you’re working with can mean you would use completely different algorithms for superficially similar situations.

Secondly, while you’re probably going to implement a lot of analytic calculations, in realistic models you’re probably going to end up using numerical methods as well. That can be because you need to work with geometry like Bézier curves or NURBS surfaces that has many freedoms. It can be because even if you start with convenient analytic shapes, new geometry that you derive from those shapes, for example by offsetting a single shape or by combining details from multiple shapes as in constructive solid geometry, won’t in general have an analytic shape itself.

By the time you allow for the numerous different types of constraint that you might want to enforce between different types of geometry and the numerous different ways you can construct new geometry from geometry you already have, the scale of the problem explodes. And on top of that, almost everything you do is going to have numerical sensitivity issues, and all but the simplest algorithms are going to need detailed, careful analysis to make sure you really have covered all the possibilities. In this field, “edge case” and “corner case” are literal terms and not just figures of speech!

To give a practical example, without looking up how to do it, could you confidently calculate whether two arbitrary cuboids are completely separate or they touch or intersect somewhere? As another example, given an arbitrary parametric surface, a sphere in a position just resting on that surface, and the constraint that the surface of the sphere must remain tangent to the parametric surface without intersecting it anywhere, how would you calculate the path the centre of the sphere will follow if you introduce gravity to start the sphere rolling in a certain direction along the surface?

These are relatively simple problems in the field, but each already has some subtlety that leaves the “obvious” solutions incomplete. Solve a few thousand problems like that, each unique and with its own calculation strategy, and now you’re starting to get a practically useful geometric modelling system. (You’ve also probably had a team of dozens of mathematicians and developers working on it for decades.)

This, IMHO, is the critical point and why a lot of “deep” development work doesn’t benefit much from the current generation of AI tools.

Last week, I was dealing with some temporal data. I often find working in this area a little frustrating because you spend so much time dealing with the inherent traps and edge cases, so using an AI code generator is superficially attractive. However, the vast majority of my time wasn’t spent writing code, it was getting my head around what the various representations of certain time-based events in this system actually mean and what should happen when they interact. I probably wrote about 100 test cases next, each covering a distinct real world scenario, and working out how to parameterise them so the coverage was exhaustive for certain tricky interactions also required a bit of thought. Finally, I wrote the implementation of this algorithm that had a lot of essential complexity, which means code with lots of conditionals that needs to be crystal clear about why things are being done in a certain order and decisions made a certain way, so anyone reading it later has a fighting chance of understanding it. Which of those three stages would current AI tools really have helped with?

I find AI code generators can be quite helpful for low-level boilerplate stuff, where the required behaviour is obvious and the details tend to be a specific database schema or remote API spec. No doubt some applications consist almost entirely of this kind of code, and I can easily believe that people working on those find AI coding tools much more effective than I typically do. But as 'manoDev says in the parent comment, deeper work is often a specification problem. The valuable part is often figuring out the what and the why rather than the how, and so far that isn’t something AI has been very good at.