ps: since when was complexity ever a good defense for why your codebase is difficult to work in?
ps: since when was complexity ever a good defense for why your codebase is difficult to work in?
A colleague of mine had the observation that the problem we're seeing is a strong tendency to produce solutions which add extra, undesirable connections in the code, resulting in inordinately complex API boundaries which are hard to properly disentangle.
That's all fixable with enough effort, but it's hard to ignore that experience when somebody comes along and tells you that vibe coding really does suffice by itself, and you can just point a good model and harness at a ticket and a codebase and let it rip.
I get that people without this mental baggage will adopt LLMs more easily as they do not care about any of these and they are often not able to perceive quality. And maybe quality is something that doesn't even exist and doesn't matter much, in the end people don't care how the sausage is made and what is inside if it tastes good and nourishes them. It surprises me though how fast most "engineers" throw away their engineering principles when using these systems. So I could also say mean things like maybe these people weren't good engineers to begin with. I guess the divide is more between people that care about the underlying principles and quality of their work vs. people that care about being done and having the desired effect on the outside world through their work. For software that's more acceptable than in other engineering disciplines I guess, people won't get harmed (much) if your vibe coded SaaS app crashes or leaks your whole user database. Trust into vibe-coded software seems to be higher than I think is warranted though, which makes sense as the companies pour trillions of USD into marketing and getting people hooked on these systems. Feels a bit like the whole crypto bubble, LLM absolutists aren't that different from crypto bros in 2015 that were certain traditional finance was doomed and in five years our entire life would play out on the blockchain. I don't think it's entirely comparable though as LLMs have tons of real-world use cases. And I think it's not surprising that people have such strong opinions on them, there were similar discussions about Rust or frontend frameworks, seems in tech there are just types of people that discover something new and immediately think it will solve all of their issues and that anyone who has a different experience must be wrong, even though they are only seeing a tiny part of the whole problem space. So, glad people get value out of LLMs and I do too, but I don't get how one can use models and not see these quality issues.
Maybe one point regarding quality, even for small scripts that I write using LLMs e.g. for data analysis I have to be very careful as they will often create subtle mistakes that ruin the whole analysis. Things that are not per se wrong as the script runs and produces the desired output (which the systems are optimized and trained for, i.e. they know how to create a script that works and kind of corresponds to the prompt), but in disciplines like data science where you need to think carefully about every step of an analysis LLMs are quite dangerous as they produce convincing looking results that seem correct at every stage but are completely wrong. The only way to get that right (in my experience) is to go really slowly testing each step individually with known good inputs and outputs, giving the LLM that as a harness to work in.
I have been looking at LLM produced artefacts for a while now especially in data science and there are very cases where the models actually produced something entirely correct, at least when it's more complex than averaging some numbers or other simple things. The same goes for writing, superficially it looks good but there are often subtle inconsistencies that the model just doesn't see and that are hard to spot. And again, some people will just look at that and think "that's fine!" because they just can't see the quality issues or they don't care about them, but that doesn't make these issues disappear...
Think about it statistically, the benchmarks say clearly these models produce correct code in maybe 90-95 % of cases for most complex questions. That sounds high but given that a real-world system can consist of e.g. 100 such individual components then even a 99 % correctness rate at the level of an individual component gives you only a 36 % correctness rate for the entire system! That's also true for papers, presentations and anything else. Again, most people won't even understand this, for them something that looks correct and works is good enough.
if this is how you write to LLMs, I can easily see why you aren’t more productive over all from using them