The fact that software quality plays zero role in commercial software engineering is one of the reasons why I think LLMs will easily eat our lunch. Bugs simply don't matter.
If all the examples you can conjure are decades old*, is it any wonder that people don't really take it seriously? Software power the whole world, and yet the example of critical failure we constantly hear about is close to half a century old?
I think the more insidious thing is all the "minor" pains being inflicted by software bugs, that when summed up reach crazy level of harm. It's just diluted so less striking. But even then, it's hard to say if the alternative of not using software would have been better overall.
* maybe they've added Boeing 737 Max to the list now?
They're not ALL the examples I can conjure up. MCAS would probably be an example of a modern software bug that killed a bunch of people.
How about the 1991 failure of the Patriot missile to defend against a SCUD missile due to a software bug not accounting for clock drift, causing 28 lives lost?
Or the 2009 loss of Air France 447 where the software displayed all sorts of confusing information in what was an unreliable airspeed situation?
Old incidents are the most likely to be widely disseminated, which is why they're most likely to be discussed, but that doesn't mean that the discussion resolving around old events mean the situation isn't happening now.
Unless bug results in enormous direct financial loses like in Knight Capital, the result is the same: no one held responsible, continue business as usual.
This might be more to do with these estimates than anything.
Until then they matter a lot.
One thing that needs to be addressed is how easy it is to build moats around your users which trap them in your ecosystem. It's great from a business perspective if you can pull it off, but it's killing innovation and making users frustrated and apathetic towards technology as a whole.
LLM seem to be really good in analyzing things. I don't not trust them to produce too much but alone the ability to take a few files and bits and pieces and ask for a response with a certain direction has been transformative to my work.
Their output is only as valuable as the human using them. If they're not a security expert to begin with, they can be easily led astray and lulled into a false sense of security.
See curl, for example. Hundreds of bogus reports rejected by an expert human. One large report with valuable data, that still requires an expert human to sift through and validate.
If it's trained on our code, how would it know any better?
If it's driven by our prompts, why would we prompt it to do any better?
1. They require gigabytes to terabytes of training data.
2. A non-trivial percentage of output data is low confidence.
The first problem requires tens to hundreds of gigabytes of training data.
This first problem not only requires the slow but predictable increase in processing power and data storage capabilities that were unachievable until recently, but is also only possible because open-source software has majorly caught on, something that was hoped for but not a given early in AI development.
The second problem means that the output will be error prone, without significant procedural processing of the output data that is a lot of work to develop. I never would have thought that software writing by neural networks would be competitive, not because of effective error control, but because the entire field of software development would be so bad at what they do (https://xkcd.com/2030/) that error-prone output would be competitive.
How do you categorize "commercial software engineering"? Does a company with $100M+ ARR count? Surely you can understand the impact that deleting a production database can have on a business.
And your answer to this is "LLMs will eat our lunch" and "bugs don't matter"? Unbelievable.
Do you really think that most businesses are prepared to handle issues caused by their own bugs, let alone those caused by the software they depend on? That's nothing but fantasy. And your attitude is "they deserve it"? Get real.
But if you have a business, and don't have continuity and recovery plans for software disasters, that's like not having fire insurance on your facility.
Fire insurance (and backups/disaster recovery plans) doesn't mean there won't be disruption, but it makes the disaster survivable, whereas without it your business is probably ended.
And losing a database or a major part of one is as simple as one adminstrator accidentally running "drop database" or "delete from customers" or "rm -rf" in the wrong environment. It happens, I've helped recover from it, and it doesn't take an AI running amok to do it.
That's the gist of your argument. They're not a "serious business", therefore it's their fault. Let's not mince words.
> It happens, I've helped recover from it, and it doesn't take an AI running amok to do it.
Again, losing a database is not the issue. I don't know why you fixated on that. The issue is that most modern software is buggy and risky to use in ways that a typical business is not well equipped to handle. "AI" can only make matters worse, with users having a false sense of confidence in its output. Thinking otherwise is delusional to the point of being dangerous.
If your business depends on that data, then you take steps to protect it. Or not, at your peril.