Comment by kragen | Hacker News Reader

Comment by kragen | Hacker News Reader

Parent Full thread

i would say, rather, that the main difference is how dependent you are on the goodwill of the vendor to solve your problems, and how much you can solve yourself (or by hiring people). which is why oracle has been mostly replaced with postgres, mysql, and sqlite; why as/400 has been mostly replaced with web servers running apache and nginx; why vaxen and the like have been mostly replaced with i386 and later amd64 hardware with buses like pci and pcie; and why vax/vms has been replaced by linux

when you have business-critical systems problems only your systems vendor can solve, over time, their pricing strategy tends to converge on the following:

1. how much money does your company make?

2. hand it over.

you may not know as much about linux as the gurus dec parachuted in to patch your vms installation knew about vms, but i can guarantee that you care more about your own problems than dec did, and that counts for more. you can either learn what you need to know or, if you have more money than time, hire somebody who already knows it—at a market rate, not a monopoly-rents rate. the perverse incentives enjoyed by vendors of closed systems slowly but surely destroyed almost all of them, leaving only a few standing

I am not sure of what point you are trying to make here.

I have been in the industry since late 80s and (surely due to the kind of customers I worked with, so I am not implying that this applies to everyone and everywhere) and I have never seen anything even remotely "mission-critical" replacing Oracle or Microsoft stacks with Linux/Postgres.

I am currently working on something based on ridiculously obsolete technology (Progress) which is running on Solaris, for example. The company I am working with is concerned about the chance that Progress will just die, but replacing it with Postgres is the least of our problems. The real problem is that a full rewrite would take maybe 5 years and nobody wants to start the ball rolling.

In the meantime, we are "more than happy" to pay whatever the licensing is both for Progress and Oracle (for Solaris) and having them "on call" in case something breaks.

i am sorry that my comment was unclear, and i would be happy to explain whatever it is that you are having trouble understanding

> I have never seen anything even remotely "mission-critical" replacing Oracle or Microsoft stacks with Linux/Postgres

yeah, this only rarely happens inside a single company. mostly the way this works is that companies that run their mission-critical systems on oracle and microsoft stacks get outcompeted in the market by companies running on linux and postgres, because they can solve their own problems instead of letting their vendors decide which problems are worth solving and what their profit margins should be allowed to be

you have, i am sure, noticed that many more companies today run their mission-critical systems on linux than on solaris, though microsoft is still in the running and even in the lead in some areas

Maybe the problem is that the company or companies you have in mind operate mostly in the service business and are relative recent - i.e. dating back to the 90s at most (e.g. AirBnb).

I sincerely doubt that for automotive/manufacturing, utilities or travel (the three industries I have most experience with) linux (or any specific IT technology) has ever made the difference between surviving and being eaten alive by competitors.

In that universe IT services are a commodity. Just like Electrical Power: you cannot operate an automotice company without electrical power, but buying it from Utility Company A instead of Utility Company B does not matter at all.

For sure when an entire industry adopts a completely new technology (e.g.: IOT Shipping) they will go for whatever seems to provide the best solution for their specific case. But their bread and butter stuff (ERP, CRM, Finance...) will either be some custom legacy app (or more probably a hodge-podge of balkanized legacy apps) or will be some customized behemoth like SAP, Oracle Finance, Microsoft Dynamics.

Both SAP and Oracle products can run on Linux, granted. But in that case I think it will be a vendor-backed version of Linux, so the support will come from the same entity that provided the application stack.

it'll be interesting to see what happens, but my perspective is pretty different from yours, which is surprising to me since we're talking about an area you have intimate and deep knowledge of. i'll explain my perspective here, in the knowledge that probably the reason for our disagreement is largely things you do know and i don't, so i fully expect this to make me look very foolish. hopefully you'll be kind enough to correct my ignorance, impressive though it may be

information technology is pretty important to travel and to logistics in general. remember that ibm tpf was originally the 'airline control program', and before that, sabre, developed in collaboration with aa. and travel and logistics planning are of course among the most economically significant uses of mixed integer linear programming.

the key question is, i think, whether operational excellence in it operations is an economically significant competitive advantage. will customers switch airlines (or hotel chains, or car rental companies, or travel agencies) because of corporate incompetence at operating their data centers? how would they even know if their airline or hotel chain or travel agency sucks at resolving operational problems, has colossal security breaches due to utter incompetence, and struggles to roll out new it-enabled features?

well, we can start with travel agencies. most travel agencies have already gone out of business. orbitz, expedia, and hipmunk ate their lunch. priceline came in and obliterated the discount special ticket market, growing into booking.com. all of these so-called online travel agencies live and die by their computer systems, and large parts of their workforce consist of programmers

how about airlines?

in 02016 delta had to cancel half their flights one day because of it incompetence: https://www.cbsnews.com/news/outage-causes-massive-delays-ca...

aa (remember, the company for which modern transaction processing was developed) had a half-hour-long nationwide outage in 02018 due to it incompetence: https://www.cnbc.com/2018/07/29/american-airlines-flights-de...

twice in 02022, british airways had a not-quite-systemwide outage due to it incompetence: https://www.datacenterdynamics.com/en/news/british-airways-i... and most ominously the reporting included the phrase, 'BA has a long history of outages that have impacted passengers.'

also in 02022, westjet had a systemwide outage due to it incompetence: https://simpleflying.com/westjet-suffers-major-system-outage...

and most famously, in 02022, southwest had an enormous systemwide outage over the christmas holidays that led them to cancel 16700 flights and cost it a billion dollars: https://www.reuters.com/business/aerospace-defense/southwest...

and this doesn't affect competition much, but the usa as a whole had an hour-and-a-half outage due to it incompetence in january of last year: https://www.dallasnews.com/business/airlines/2023/01/11/wide...

later last year, spirit airlines had a systemwide outage: https://www.dailymail.co.uk/news/article-12148955/Spirit-Air...

and so did southwest, again: https://www.reuters.com/business/aerospace-defense/southwest...

mckinsey has a list of recent airline problems due to it incompetence that have affected customers, including in some cases not only delayed flights but leaks of private data customers are legally required to provide to airlines: https://www.mckinsey.com/~/media/McKinsey/Industries/Travel%... (but unfortunately they have carefully purged it of details that would make each case independently verifiable)

so i would say that specifically in the case of travel, incompetence at information technology operations is a significant public relations problem already; it regularly makes national and world news. and we should expect this to get much worse before it gets better. today the saving grace is that no airline currently has reasonably competent information technology operations, so there's no competitor eating southwest's lunch. but when your incompetence at running your computer systems loses you a billion dollars here and a billion dollars there, sooner or later it starts to add up to real money

with respect to manufacturing, i think it's well known that jit supply chain management is extremely demanding on it operations, and since covid began, supply-chain disruption has been an enormous stress on the profitability of manufacturing operations—in particular, in the automotive sector

more broadly, though, increasing automation of manufacturing is a crucial competitive advantage; to take one example, today jlcpcb or pcbway will sell you custom pcb manufacturing services for fifty dollars (or two dollars as a promotional offer) that would cost you a thousand dollars and up from flextronics. and now they're moving into cnc machining, as well, and of course service bureaus like shapeways have been around for a while, gradually moving up the value chain. all of this is crucially dependent on the operational stability of their web sites and their resistance to data breaches and ransomware, but so far they're doing a lot better than the airlines

this is not just a big challenge for traditional job shops, who are at risk of being ghettoized into a constantly shrinking super-premium market as disruptive innovation eats them from below; it's also a huge opportunity for new kinds of products that depend on the low-cost availability of such highly automated custom manufacturing for their profitability

in a related development, many high-value, highly complex products manufactured using traditional techniques—most obviously mechanical wristwatches, cameras, and various kinds of on-orbit systems, but also timers, governors, carburetors, power transmission systems, internal combustion engines, and now even lightswitches and lightbulbs—are being replaced by mechanically much simpler systems which instead rely heavily on embedded software and, in some cases, centralized cloud services and much more highly automated manufacturing

having your car start reliably, and your steering and braking systems work reliably, and being able to turn on the heater without taking your eyes off the road, hasn't been much of a competitive advantage for decades; but, as much of the unreliability of those systems starts to stem from software incompetence instead of incompetence at hydraulics and gears, we should start to see automotive suppliers competing not just on how efficient their inventory management is but on how reliable and featureful and usable their firmware is. at the most prosaic level, if the bluetooth audio only works half the time you get into your friend's chevrolet, you're more likely to buy a hyundai

utilities, though, i agree. that's because utilities are regulated monopolies, so it doesn't matter how incompetent they are; relatively few companies and fewer people are going to move out of texas because ercot and its member utilities can't mitigate even the most glaring risks to system stability. pg&e might be an exception, because its sky-high energy prices are making california heavy industry as a whole economically uncompetitive on the world scene, but pg&e isn't going to go bankrupt; its customers are

fundamentally, though, in the 20th century companies were operated by managers, and in the 21st century, increasingly, they are operated by computers—as are physical products. outsourcing your computing makes as much sense as outsourcing your management

You make a long list of things that you probably had to google on, but you still seem to lack any actual, hands-on experience on how things actually work and mostly important why systems like SABRE take forever to be replaced. Trust me: it was not because of Linux or the lack of it.

Most importantly, though, you keep missing the forest for the three: yes, modern cars have more and more electronics in them, and this most probably runs on customized, hardened Linux or some other modern OS. And they need constant communication with cloud based services etc. etc. This applies also to fridges, TVs and so on.

My point is that IN ORDER TO PRODUCE THE CAR, FRIDGE OR TV manufacturing industry still uses for the most part legacy systems and that the cost to replace these with newly developed applications (that for some reason are either available only on Linux or "work better" on Linux) is never been appealing enough to justify the switchover.

[also, please stop quoting buzzwords at me: "JIT supply chain management" is what I was working on when I encountered the aforementioned problem on DEX/VAX. The application was written in COBOL, but was the first project where we adopted Oracle (Version 6 I believe) instead of a Hierarchical DBMS. The year was 1991 or 1992, and the company I was working for was definitely a late adopter].

When companies producing TVs switched to more modern, larger screens (let's say LCD/LED)... they just changed the BOM of the new models in their own heavily customized SAP instance. Maybe they had to add a couple of fields to a table because of new characteristics they needed to track and that were not present on older models. They probably revamped their production lines, for sure. They had to hire younger engineers to develop or productivize (sp?) these new technologies. This still did not make a dent in the large pile of legacy code that was used to manage the company as a whole.

Even when the PRODUCT is totally, radically new, the systems that manage its production (and sale, and post sale support) do not need to follow suit. You still need to order components to suppliers, schedule the production, deliver the completed product to the customer or to the reseller, possibly keep track of where something was sold/sent in case they need to recall some faulty batch or police needs to know about it (a common requirements for vehicles, for example).

The only exceptions to this simple rule are companies like Tesla, i.e. someone who creates a completely new manufacturing company from scratch. Of course, these are outliers, but they are also the only ones that don't already have an enormous amount of resources invested in systems created 25+ years ago.

The other exceptions, of course, are companies who do not produce or handle anything physical (remember when I mentioned Airbnb?). These, too, can start from scratch, and maybe for these the technology makes a difference. On the other hand, a startup cannot really afford to pay the exorbitant fees requested by Oracle, and they find it much more convenient using Postgres or any other Open Source product.

More power to them. All this does not change my original viewpoint:

IT for mature companies is exactly like any other utility. If you need fuel for your steel mill furnace, changing provider every 3 months might help you save a few thousands bucks (because you use so much of the stuff that even a sub-cent decrease in price will be noticeable) ... but by doing so you will probably have to pay 10 times that just guarantee that the switchover to the new supplier goes on without any glitch (like having your gas supply cut off one day or two before the new one starts working). If you need electrical power you may consider to put solar panels on the roof of your plants... just like you may want to put a REST interface in front of some of your stored procedures. But you will still depend on the power company for 99% of your energy, and you will still have to mantain your stored procedure because if you need to change something there, no amount of JSON will solve the problem.

This is the gist of my argument, if it still sounds unconvincing to you it may just be because I worked most of my career server side as a system integration expert for large, mature companies. Maybe you have a different background - companies established after the web was born are surely a completely different beast, and my experience most probably would not apply, but as long as we are talking of anything founded before 1990 I am pretty sure that inertia is the prevalent force that shaped their IT landscape.

i can't find much to disagree with in your comment. i wish i could disagree with the snarky comments about my lack of expertise, but unfortunately those are on target too :)

but it sounds like you think some things about my previous comment are wrong; i just can't tell what any of them are

Sorry: I checked your cv and I can assure that your skills as a developer are at least 10x what I had even at the top of my career. I type mostly from my phone so I am often hasty (and make lots of typos). I assure you that when maybe looked like "lack of expertise" I meant "less experience dealing with the messy underbelly of large, mature companies".

Apologies for the tone anyway.

while i appreciate the groundless flattery, no apologies are necessary; whatever effects the tone may have on my ego are less important than your generosity in sharing your experience

Reply on news.ycombinator.com