Except it was not AS400 or anything hardware... it was a problem with an Oracle product, and took a bit over one week with higher and higher gurus being parachuted in by Oracle Europe.
And in the early 90s I had another similar episode, this time with DEC/Vax. Not sure anymore but I think they finally had to patch the OS.
My point: nothing made by humans can guarantee 101% uptime. The main difference is how far any vendor will go to solve your problems.
when you have business-critical systems problems only your systems vendor can solve, over time, their pricing strategy tends to converge on the following:
1. how much money does your company make?
2. hand it over.
you may not know as much about linux as the gurus dec parachuted in to patch your vms installation knew about vms, but i can guarantee that you care more about your own problems than dec did, and that counts for more. you can either learn what you need to know or, if you have more money than time, hire somebody who already knows it—at a market rate, not a monopoly-rents rate. the perverse incentives enjoyed by vendors of closed systems slowly but surely destroyed almost all of them, leaving only a few standing
I have been in the industry since late 80s and (surely due to the kind of customers I worked with, so I am not implying that this applies to everyone and everywhere) and I have never seen anything even remotely "mission-critical" replacing Oracle or Microsoft stacks with Linux/Postgres.
I am currently working on something based on ridiculously obsolete technology (Progress) which is running on Solaris, for example. The company I am working with is concerned about the chance that Progress will just die, but replacing it with Postgres is the least of our problems. The real problem is that a full rewrite would take maybe 5 years and nobody wants to start the ball rolling.
In the meantime, we are "more than happy" to pay whatever the licensing is both for Progress and Oracle (for Solaris) and having them "on call" in case something breaks.
> I have never seen anything even remotely "mission-critical" replacing Oracle or Microsoft stacks with Linux/Postgres
yeah, this only rarely happens inside a single company. mostly the way this works is that companies that run their mission-critical systems on oracle and microsoft stacks get outcompeted in the market by companies running on linux and postgres, because they can solve their own problems instead of letting their vendors decide which problems are worth solving and what their profit margins should be allowed to be
you have, i am sure, noticed that many more companies today run their mission-critical systems on linux than on solaris, though microsoft is still in the running and even in the lead in some areas
I sincerely doubt that for automotive/manufacturing, utilities or travel (the three industries I have most experience with) linux (or any specific IT technology) has ever made the difference between surviving and being eaten alive by competitors.
In that universe IT services are a commodity. Just like Electrical Power: you cannot operate an automotice company without electrical power, but buying it from Utility Company A instead of Utility Company B does not matter at all.
For sure when an entire industry adopts a completely new technology (e.g.: IOT Shipping) they will go for whatever seems to provide the best solution for their specific case. But their bread and butter stuff (ERP, CRM, Finance...) will either be some custom legacy app (or more probably a hodge-podge of balkanized legacy apps) or will be some customized behemoth like SAP, Oracle Finance, Microsoft Dynamics.
Both SAP and Oracle products can run on Linux, granted. But in that case I think it will be a vendor-backed version of Linux, so the support will come from the same entity that provided the application stack.
information technology is pretty important to travel and to logistics in general. remember that ibm tpf was originally the 'airline control program', and before that, sabre, developed in collaboration with aa. and travel and logistics planning are of course among the most economically significant uses of mixed integer linear programming.
the key question is, i think, whether operational excellence in it operations is an economically significant competitive advantage. will customers switch airlines (or hotel chains, or car rental companies, or travel agencies) because of corporate incompetence at operating their data centers? how would they even know if their airline or hotel chain or travel agency sucks at resolving operational problems, has colossal security breaches due to utter incompetence, and struggles to roll out new it-enabled features?
well, we can start with travel agencies. most travel agencies have already gone out of business. orbitz, expedia, and hipmunk ate their lunch. priceline came in and obliterated the discount special ticket market, growing into booking.com. all of these so-called online travel agencies live and die by their computer systems, and large parts of their workforce consist of programmers
how about airlines?
in 02016 delta had to cancel half their flights one day because of it incompetence: https://www.cbsnews.com/news/outage-causes-massive-delays-ca...
aa (remember, the company for which modern transaction processing was developed) had a half-hour-long nationwide outage in 02018 due to it incompetence: https://www.cnbc.com/2018/07/29/american-airlines-flights-de...
twice in 02022, british airways had a not-quite-systemwide outage due to it incompetence: https://www.datacenterdynamics.com/en/news/british-airways-i... and most ominously the reporting included the phrase, 'BA has a long history of outages that have impacted passengers.'
also in 02022, westjet had a systemwide outage due to it incompetence: https://simpleflying.com/westjet-suffers-major-system-outage...
and most famously, in 02022, southwest had an enormous systemwide outage over the christmas holidays that led them to cancel 16700 flights and cost it a billion dollars: https://www.reuters.com/business/aerospace-defense/southwest...
and this doesn't affect competition much, but the usa as a whole had an hour-and-a-half outage due to it incompetence in january of last year: https://www.dallasnews.com/business/airlines/2023/01/11/wide...
later last year, spirit airlines had a systemwide outage: https://www.dailymail.co.uk/news/article-12148955/Spirit-Air...
and so did southwest, again: https://www.reuters.com/business/aerospace-defense/southwest...
mckinsey has a list of recent airline problems due to it incompetence that have affected customers, including in some cases not only delayed flights but leaks of private data customers are legally required to provide to airlines: https://www.mckinsey.com/~/media/McKinsey/Industries/Travel%... (but unfortunately they have carefully purged it of details that would make each case independently verifiable)
so i would say that specifically in the case of travel, incompetence at information technology operations is a significant public relations problem already; it regularly makes national and world news. and we should expect this to get much worse before it gets better. today the saving grace is that no airline currently has reasonably competent information technology operations, so there's no competitor eating southwest's lunch. but when your incompetence at running your computer systems loses you a billion dollars here and a billion dollars there, sooner or later it starts to add up to real money
with respect to manufacturing, i think it's well known that jit supply chain management is extremely demanding on it operations, and since covid began, supply-chain disruption has been an enormous stress on the profitability of manufacturing operations—in particular, in the automotive sector
more broadly, though, increasing automation of manufacturing is a crucial competitive advantage; to take one example, today jlcpcb or pcbway will sell you custom pcb manufacturing services for fifty dollars (or two dollars as a promotional offer) that would cost you a thousand dollars and up from flextronics. and now they're moving into cnc machining, as well, and of course service bureaus like shapeways have been around for a while, gradually moving up the value chain. all of this is crucially dependent on the operational stability of their web sites and their resistance to data breaches and ransomware, but so far they're doing a lot better than the airlines
this is not just a big challenge for traditional job shops, who are at risk of being ghettoized into a constantly shrinking super-premium market as disruptive innovation eats them from below; it's also a huge opportunity for new kinds of products that depend on the low-cost availability of such highly automated custom manufacturing for their profitability
in a related development, many high-value, highly complex products manufactured using traditional techniques—most obviously mechanical wristwatches, cameras, and various kinds of on-orbit systems, but also timers, governors, carburetors, power transmission systems, internal combustion engines, and now even lightswitches and lightbulbs—are being replaced by mechanically much simpler systems which instead rely heavily on embedded software and, in some cases, centralized cloud services and much more highly automated manufacturing
having your car start reliably, and your steering and braking systems work reliably, and being able to turn on the heater without taking your eyes off the road, hasn't been much of a competitive advantage for decades; but, as much of the unreliability of those systems starts to stem from software incompetence instead of incompetence at hydraulics and gears, we should start to see automotive suppliers competing not just on how efficient their inventory management is but on how reliable and featureful and usable their firmware is. at the most prosaic level, if the bluetooth audio only works half the time you get into your friend's chevrolet, you're more likely to buy a hyundai
utilities, though, i agree. that's because utilities are regulated monopolies, so it doesn't matter how incompetent they are; relatively few companies and fewer people are going to move out of texas because ercot and its member utilities can't mitigate even the most glaring risks to system stability. pg&e might be an exception, because its sky-high energy prices are making california heavy industry as a whole economically uncompetitive on the world scene, but pg&e isn't going to go bankrupt; its customers are
fundamentally, though, in the 20th century companies were operated by managers, and in the 21st century, increasingly, they are operated by computers—as are physical products. outsourcing your computing makes as much sense as outsourcing your management
Most importantly, though, you keep missing the forest for the three: yes, modern cars have more and more electronics in them, and this most probably runs on customized, hardened Linux or some other modern OS. And they need constant communication with cloud based services etc. etc. This applies also to fridges, TVs and so on.
My point is that IN ORDER TO PRODUCE THE CAR, FRIDGE OR TV manufacturing industry still uses for the most part legacy systems and that the cost to replace these with newly developed applications (that for some reason are either available only on Linux or "work better" on Linux) is never been appealing enough to justify the switchover.
[also, please stop quoting buzzwords at me: "JIT supply chain management" is what I was working on when I encountered the aforementioned problem on DEX/VAX. The application was written in COBOL, but was the first project where we adopted Oracle (Version 6 I believe) instead of a Hierarchical DBMS. The year was 1991 or 1992, and the company I was working for was definitely a late adopter].
When companies producing TVs switched to more modern, larger screens (let's say LCD/LED)... they just changed the BOM of the new models in their own heavily customized SAP instance. Maybe they had to add a couple of fields to a table because of new characteristics they needed to track and that were not present on older models. They probably revamped their production lines, for sure. They had to hire younger engineers to develop or productivize (sp?) these new technologies. This still did not make a dent in the large pile of legacy code that was used to manage the company as a whole.
Even when the PRODUCT is totally, radically new, the systems that manage its production (and sale, and post sale support) do not need to follow suit. You still need to order components to suppliers, schedule the production, deliver the completed product to the customer or to the reseller, possibly keep track of where something was sold/sent in case they need to recall some faulty batch or police needs to know about it (a common requirements for vehicles, for example).
The only exceptions to this simple rule are companies like Tesla, i.e. someone who creates a completely new manufacturing company from scratch. Of course, these are outliers, but they are also the only ones that don't already have an enormous amount of resources invested in systems created 25+ years ago.
The other exceptions, of course, are companies who do not produce or handle anything physical (remember when I mentioned Airbnb?). These, too, can start from scratch, and maybe for these the technology makes a difference. On the other hand, a startup cannot really afford to pay the exorbitant fees requested by Oracle, and they find it much more convenient using Postgres or any other Open Source product.
More power to them. All this does not change my original viewpoint:
IT for mature companies is exactly like any other utility. If you need fuel for your steel mill furnace, changing provider every 3 months might help you save a few thousands bucks (because you use so much of the stuff that even a sub-cent decrease in price will be noticeable) ... but by doing so you will probably have to pay 10 times that just guarantee that the switchover to the new supplier goes on without any glitch (like having your gas supply cut off one day or two before the new one starts working). If you need electrical power you may consider to put solar panels on the roof of your plants... just like you may want to put a REST interface in front of some of your stored procedures. But you will still depend on the power company for 99% of your energy, and you will still have to mantain your stored procedure because if you need to change something there, no amount of JSON will solve the problem.
This is the gist of my argument, if it still sounds unconvincing to you it may just be because I worked most of my career server side as a system integration expert for large, mature companies. Maybe you have a different background - companies established after the web was born are surely a completely different beast, and my experience most probably would not apply, but as long as we are talking of anything founded before 1990 I am pretty sure that inertia is the prevalent force that shaped their IT landscape.
but it sounds like you think some things about my previous comment are wrong; i just can't tell what any of them are
Apologies for the tone anyway.
From my understanding that was basically Oracle's business model back in the day (or the model of a lot of employees). Extensive consultant billable hours for maintenance and integration after you buy it.
From my experience, IBM will go to the far side of the moon and back in order to solve your problem. It will probably cost you in fees as much as buying a new car, but there's no luck of resources or resilience from their part. And I admire them for that. As much as we like to idolize the latest and greatest in this forum, the truth is that when it comes to mission critical equipment you need someone that you know will fly an expert from a different country in the middle of the night if they have to.
Something similar happened to a client a couple of years ago. They wanted to set up their "devops pipeline" on AWS.
First they tried to solve it in house. After two months they gave up and brought in an AWS specialist. After two weeks, we had two more AWS specialist inclduing one really expensive. 1.5 months later we had the pipeline going, but fragile as hell.
I have not been able to piece together how it could have possibly taken so long. I didn not work on this part of the project myself at all.
Given that the entire back-edn was in C#, with VSProject configurations the whole bit, it would have taken less than week on Azure, if you kept some of the presets.
But yeah Azure CI/CD can suck as well and its less configurable. Which is a curse or a benefit depending on your standpoint.