Satya Nadella uses an IBM AS/400 in 1993
thechipletter.substack.com
thechipletter.substack.com
I am not saying it is the most reliable server in the world, because there are a lot I have not worked with, but it was far more reliable than anything else I have worked on.
IF something did go wrong, they would at times call IBM and ask for service before the customer had any knowledge of an issue.
My mother became the AS/400 sys admin at her company and she had no idea about servers or technical sides of computers. She changed the backup tapes on a schedule and that was it .
Many places nobody knew where the box was since nobody interacted with it on a physical level. Green screen terminals or terminal apps on Windows was the norm.
Some business had them for inventory / sales / accounting and so on.
Many places nobody knew where the box was since nobody interacted with it on a physical level. Green screen terminals or terminal apps on Windows was the norm.
There were a couple of simple games (not from IBM) you could download and play, but it wasn't worth it.
There were some drawbacks, but Id love to have one to play with.
It's unlikely that anything under a 14nm process node is used for the CPU, if that.
IBM has deep knowledge in high-availability systems, but it's unlikely that you could call these more advanced than a modern 5nm Ryzen server.
Performance between junior basic code and heavily optimized complex queries even on just Oracle can be 1000x easily, just ask any data warehouse guy (seen it few times myself even though I don't do warehouses). You can maybe add another 0 for AS-400 there in extreme cases.
Yes if you end up doing things wrong and expect that CPU to do some heavy math then yes this will be dusted quickly. Otherwise, not so much.
But this goes against most modern 'SV principles' so I don't expect much love from younger generations here. Business love that though, and secretly wish all IT could be as reliable and predictable as them although that ship has sailed long time ago. One of those 'they don't do them like good old times anymore'.
You might also find it interesting to consider that the CPU was likely made by Global Foundries (assuming that their arrangement with IBM is still in place).
I am still trying to get my coworkers excited about VMS on x86; our VAX users are completely disinterested.
I have my own skunkworks VMS 1.0 on SimH, from these instructions:
Naturally NGEN has the caveat of only being good enough for faster startup, and requiring strong binaries (aka signed dlls), which meant not everyone adopted it.
Windows 8.x adopted Bartok from Singularity, where applications would be precompiled on the store, and linked on device at instalation time.
Windows 10 moved full AOT compilation into the store when downloading into specific devices.
Note that full AOT on Android is only on versions 5 and 6, starting with version 7 onwards is a mix of interpretation, JIT and AOT, which is nonetheless quite cool.
Also there are several optimizations that are only possible after having PGO data, and that was lost on AOT at installation time.
We had massive queries that ran in seconds that HANA with millions blown into the consultants just couldn’t complete in time (nightly billing runs).
The solution in the end was that a lot of complexity and decades of cleverness in adjusting to specific patterns in customer behavior were thrown out and replaced by “the consultant will just adjust that every week by hand”.
Also - I was soooo much faster with a num pad and the terminal than clicking around the SAP GUI. Ultimately part of why I left there.
/no experience with SAP but I’ve never heard a successful story implementing it ever
OP isn't saying the SAP implementation wasn't successful so, just expensive (no surprise), and somethings didn't work as begore (no surprise neither).
It also has a ton of RAS features.[0]
ECC beyond SECDED, processor instruction retry, CRC on all fabrics part of which can fail and the system will degrade to half-bandwidth. Hot/Swap and/or Redudant everything to include the LCD panel on the OP Panel.
I'm actually curious what type of RAS stuff EPYC and XEON have, and hope somebody can link to the info.
Also had a Domino mail server (for lotus notes. It was way ahead of its time (at the time) and also super rock solid. Neither ever gave me any trouble... even when the Domino server ran out of disk space.. it never died or corrupted. just let me know and when i fixed it it carried on like there was no issue.
Except it was not AS400 or anything hardware... it was a problem with an Oracle product, and took a bit over one week with higher and higher gurus being parachuted in by Oracle Europe.
And in the early 90s I had another similar episode, this time with DEC/Vax. Not sure anymore but I think they finally had to patch the OS.
My point: nothing made by humans can guarantee 101% uptime. The main difference is how far any vendor will go to solve your problems.
when you have business-critical systems problems only your systems vendor can solve, over time, their pricing strategy tends to converge on the following:
1. how much money does your company make?
2. hand it over.
you may not know as much about linux as the gurus dec parachuted in to patch your vms installation knew about vms, but i can guarantee that you care more about your own problems than dec did, and that counts for more. you can either learn what you need to know or, if you have more money than time, hire somebody who already knows it—at a market rate, not a monopoly-rents rate. the perverse incentives enjoyed by vendors of closed systems slowly but surely destroyed almost all of them, leaving only a few standing
I have been in the industry since late 80s and (surely due to the kind of customers I worked with, so I am not implying that this applies to everyone and everywhere) and I have never seen anything even remotely "mission-critical" replacing Oracle or Microsoft stacks with Linux/Postgres.
I am currently working on something based on ridiculously obsolete technology (Progress) which is running on Solaris, for example. The company I am working with is concerned about the chance that Progress will just die, but replacing it with Postgres is the least of our problems. The real problem is that a full rewrite would take maybe 5 years and nobody wants to start the ball rolling.
In the meantime, we are "more than happy" to pay whatever the licensing is both for Progress and Oracle (for Solaris) and having them "on call" in case something breaks.
> I have never seen anything even remotely "mission-critical" replacing Oracle or Microsoft stacks with Linux/Postgres
yeah, this only rarely happens inside a single company. mostly the way this works is that companies that run their mission-critical systems on oracle and microsoft stacks get outcompeted in the market by companies running on linux and postgres, because they can solve their own problems instead of letting their vendors decide which problems are worth solving and what their profit margins should be allowed to be
you have, i am sure, noticed that many more companies today run their mission-critical systems on linux than on solaris, though microsoft is still in the running and even in the lead in some areas
I sincerely doubt that for automotive/manufacturing, utilities or travel (the three industries I have most experience with) linux (or any specific IT technology) has ever made the difference between surviving and being eaten alive by competitors.
In that universe IT services are a commodity. Just like Electrical Power: you cannot operate an automotice company without electrical power, but buying it from Utility Company A instead of Utility Company B does not matter at all.
For sure when an entire industry adopts a completely new technology (e.g.: IOT Shipping) they will go for whatever seems to provide the best solution for their specific case. But their bread and butter stuff (ERP, CRM, Finance...) will either be some custom legacy app (or more probably a hodge-podge of balkanized legacy apps) or will be some customized behemoth like SAP, Oracle Finance, Microsoft Dynamics.
Both SAP and Oracle products can run on Linux, granted. But in that case I think it will be a vendor-backed version of Linux, so the support will come from the same entity that provided the application stack.
information technology is pretty important to travel and to logistics in general. remember that ibm tpf was originally the 'airline control program', and before that, sabre, developed in collaboration with aa. and travel and logistics planning are of course among the most economically significant uses of mixed integer linear programming.
the key question is, i think, whether operational excellence in it operations is an economically significant competitive advantage. will customers switch airlines (or hotel chains, or car rental companies, or travel agencies) because of corporate incompetence at operating their data centers? how would they even know if their airline or hotel chain or travel agency sucks at resolving operational problems, has colossal security breaches due to utter incompetence, and struggles to roll out new it-enabled features?
well, we can start with travel agencies. most travel agencies have already gone out of business. orbitz, expedia, and hipmunk ate their lunch. priceline came in and obliterated the discount special ticket market, growing into booking.com. all of these so-called online travel agencies live and die by their computer systems, and large parts of their workforce consist of programmers
how about airlines?
in 02016 delta had to cancel half their flights one day because of it incompetence: https://www.cbsnews.com/news/outage-causes-massive-delays-ca...
aa (remember, the company for which modern transaction processing was developed) had a half-hour-long nationwide outage in 02018 due to it incompetence: https://www.cnbc.com/2018/07/29/american-airlines-flights-de...
twice in 02022, british airways had a not-quite-systemwide outage due to it incompetence: https://www.datacenterdynamics.com/en/news/british-airways-i... and most ominously the reporting included the phrase, 'BA has a long history of outages that have impacted passengers.'
also in 02022, westjet had a systemwide outage due to it incompetence: https://simpleflying.com/westjet-suffers-major-system-outage...
and most famously, in 02022, southwest had an enormous systemwide outage over the christmas holidays that led them to cancel 16700 flights and cost it a billion dollars: https://www.reuters.com/business/aerospace-defense/southwest...
and this doesn't affect competition much, but the usa as a whole had an hour-and-a-half outage due to it incompetence in january of last year: https://www.dallasnews.com/business/airlines/2023/01/11/wide...
later last year, spirit airlines had a systemwide outage: https://www.dailymail.co.uk/news/article-12148955/Spirit-Air...
and so did southwest, again: https://www.reuters.com/business/aerospace-defense/southwest...
mckinsey has a list of recent airline problems due to it incompetence that have affected customers, including in some cases not only delayed flights but leaks of private data customers are legally required to provide to airlines: https://www.mckinsey.com/~/media/McKinsey/Industries/Travel%... (but unfortunately they have carefully purged it of details that would make each case independently verifiable)
so i would say that specifically in the case of travel, incompetence at information technology operations is a significant public relations problem already; it regularly makes national and world news. and we should expect this to get much worse before it gets better. today the saving grace is that no airline currently has reasonably competent information technology operations, so there's no competitor eating southwest's lunch. but when your incompetence at running your computer systems loses you a billion dollars here and a billion dollars there, sooner or later it starts to add up to real money
with respect to manufacturing, i think it's well known that jit supply chain management is extremely demanding on it operations, and since covid began, supply-chain disruption has been an enormous stress on the profitability of manufacturing operations—in particular, in the automotive sector
more broadly, though, increasing automation of manufacturing is a crucial competitive advantage; to take one example, today jlcpcb or pcbway will sell you custom pcb manufacturing services for fifty dollars (or two dollars as a promotional offer) that would cost you a thousand dollars and up from flextronics. and now they're moving into cnc machining, as well, and of course service bureaus like shapeways have been around for a while, gradually moving up the value chain. all of this is crucially dependent on the operational stability of their web sites and their resistance to data breaches and ransomware, but so far they're doing a lot better than the airlines
this is not just a big challenge for traditional job shops, who are at risk of being ghettoized into a constantly shrinking super-premium market as disruptive innovation eats them from below; it's also a huge opportunity for new kinds of products that depend on the low-cost availability of such highly automated custom manufacturing for their profitability
in a related development, many high-value, highly complex products manufactured using traditional techniques—most obviously mechanical wristwatches, cameras, and various kinds of on-orbit systems, but also timers, governors, carburetors, power transmission systems, internal combustion engines, and now even lightswitches and lightbulbs—are being replaced by mechanically much simpler systems which instead rely heavily on embedded software and, in some cases, centralized cloud services and much more highly automated manufacturing
having your car start reliably, and your steering and braking systems work reliably, and being able to turn on the heater without taking your eyes off the road, hasn't been much of a competitive advantage for decades; but, as much of the unreliability of those systems starts to stem from software incompetence instead of incompetence at hydraulics and gears, we should start to see automotive suppliers competing not just on how efficient their inventory management is but on how reliable and featureful and usable their firmware is. at the most prosaic level, if the bluetooth audio only works half the time you get into your friend's chevrolet, you're more likely to buy a hyundai
utilities, though, i agree. that's because utilities are regulated monopolies, so it doesn't matter how incompetent they are; relatively few companies and fewer people are going to move out of texas because ercot and its member utilities can't mitigate even the most glaring risks to system stability. pg&e might be an exception, because its sky-high energy prices are making california heavy industry as a whole economically uncompetitive on the world scene, but pg&e isn't going to go bankrupt; its customers are
fundamentally, though, in the 20th century companies were operated by managers, and in the 21st century, increasingly, they are operated by computers—as are physical products. outsourcing your computing makes as much sense as outsourcing your management
Most importantly, though, you keep missing the forest for the three: yes, modern cars have more and more electronics in them, and this most probably runs on customized, hardened Linux or some other modern OS. And they need constant communication with cloud based services etc. etc. This applies also to fridges, TVs and so on.
My point is that IN ORDER TO PRODUCE THE CAR, FRIDGE OR TV manufacturing industry still uses for the most part legacy systems and that the cost to replace these with newly developed applications (that for some reason are either available only on Linux or "work better" on Linux) is never been appealing enough to justify the switchover.
[also, please stop quoting buzzwords at me: "JIT supply chain management" is what I was working on when I encountered the aforementioned problem on DEX/VAX. The application was written in COBOL, but was the first project where we adopted Oracle (Version 6 I believe) instead of a Hierarchical DBMS. The year was 1991 or 1992, and the company I was working for was definitely a late adopter].
When companies producing TVs switched to more modern, larger screens (let's say LCD/LED)... they just changed the BOM of the new models in their own heavily customized SAP instance. Maybe they had to add a couple of fields to a table because of new characteristics they needed to track and that were not present on older models. They probably revamped their production lines, for sure. They had to hire younger engineers to develop or productivize (sp?) these new technologies. This still did not make a dent in the large pile of legacy code that was used to manage the company as a whole.
Even when the PRODUCT is totally, radically new, the systems that manage its production (and sale, and post sale support) do not need to follow suit. You still need to order components to suppliers, schedule the production, deliver the completed product to the customer or to the reseller, possibly keep track of where something was sold/sent in case they need to recall some faulty batch or police needs to know about it (a common requirements for vehicles, for example).
The only exceptions to this simple rule are companies like Tesla, i.e. someone who creates a completely new manufacturing company from scratch. Of course, these are outliers, but they are also the only ones that don't already have an enormous amount of resources invested in systems created 25+ years ago.
The other exceptions, of course, are companies who do not produce or handle anything physical (remember when I mentioned Airbnb?). These, too, can start from scratch, and maybe for these the technology makes a difference. On the other hand, a startup cannot really afford to pay the exorbitant fees requested by Oracle, and they find it much more convenient using Postgres or any other Open Source product.
More power to them. All this does not change my original viewpoint:
IT for mature companies is exactly like any other utility. If you need fuel for your steel mill furnace, changing provider every 3 months might help you save a few thousands bucks (because you use so much of the stuff that even a sub-cent decrease in price will be noticeable) ... but by doing so you will probably have to pay 10 times that just guarantee that the switchover to the new supplier goes on without any glitch (like having your gas supply cut off one day or two before the new one starts working). If you need electrical power you may consider to put solar panels on the roof of your plants... just like you may want to put a REST interface in front of some of your stored procedures. But you will still depend on the power company for 99% of your energy, and you will still have to mantain your stored procedure because if you need to change something there, no amount of JSON will solve the problem.
This is the gist of my argument, if it still sounds unconvincing to you it may just be because I worked most of my career server side as a system integration expert for large, mature companies. Maybe you have a different background - companies established after the web was born are surely a completely different beast, and my experience most probably would not apply, but as long as we are talking of anything founded before 1990 I am pretty sure that inertia is the prevalent force that shaped their IT landscape.
but it sounds like you think some things about my previous comment are wrong; i just can't tell what any of them are
Apologies for the tone anyway.
From my understanding that was basically Oracle's business model back in the day (or the model of a lot of employees). Extensive consultant billable hours for maintenance and integration after you buy it.
From my experience, IBM will go to the far side of the moon and back in order to solve your problem. It will probably cost you in fees as much as buying a new car, but there's no luck of resources or resilience from their part. And I admire them for that. As much as we like to idolize the latest and greatest in this forum, the truth is that when it comes to mission critical equipment you need someone that you know will fly an expert from a different country in the middle of the night if they have to.
Something similar happened to a client a couple of years ago. They wanted to set up their "devops pipeline" on AWS.
First they tried to solve it in house. After two months they gave up and brought in an AWS specialist. After two weeks, we had two more AWS specialist inclduing one really expensive. 1.5 months later we had the pipeline going, but fragile as hell.
I have not been able to piece together how it could have possibly taken so long. I didn not work on this part of the project myself at all.
Given that the entire back-edn was in C#, with VSProject configurations the whole bit, it would have taken less than week on Azure, if you kept some of the presets.
But yeah Azure CI/CD can suck as well and its less configurable. Which is a curse or a benefit depending on your standpoint.
One of the shops I worked at had an AS/400 and related equipment sitting on a pallet by the elevator bank. I could have had it for free.
Drawbacks? I had no way to transport a pallet of computer equipment. And IIRC it required 3 phase power so care and feeding would have been well beyond my means. (But I still have and occasionally use the IBM Model M that they were going to discard and another time I carried a retired Sun pizza box home on the train.)
I always assumed their call home service was triggered by SNMP but may be wrong? Regardless, IBM exposed a lot of metrics via SNMP so it was always an easy way to query it for metrics and/or accept traps from their devices/OS’s for failures
One of the great features of AS/400 is I think shift+esc, which allows you to quickly view the list of tables (or files as they call them) that are being used to populate the current screen. This should be a standard function for Tableau/Power BI workbooks that have live database connections.
I have a love-hate relationship with it. It's like an old pickup with 1M miles that refuses to die.
I am not sure if it is a requirement when creating a table to have descriptions but they are all there in my org. So go to the screen to get the tables and then join it with a "give me the list of all tables, table descriptions, columns, column descriptions" SQL query. EZ.
After seeing how useful these descriptions are, I firmly believe that every organization should make them a requirement even if the rdb itself does not. Just simple form of documentation at time of creation that allows for context while still giving you the ability to enforce rigid naming structures.
Surely things have gotten faster and (in some cases!) more efficient, but we are doing the same thing 30 years later that we were excited about in 1993. In a way that’s both comforting and darkly funny.
I think I've worked on the problem of "we want our sales guys to get an email whenever the sale they made ships" ... an un-countable amount of times. Without fail the first and biggest hurdle is that the customer does not reliably capture or connect their sales people's email addresses in any kind of reliable way. I'm sure they do when it comes to paying bonuses, but not for sending the email when the thing ships. It makes no sense, it should be easy, but they don't. So we work on it on and on and then with the next customer and so on.
And honestly I'm not sure any of these sales folks read the dang email.
> The 1916 Type 53 was the first car to use the same control layout as modern automobiles- with the gear lever and hand brake in the middle of the front two seats, a key started ignition, and three pedals for the clutch, brake and throttle in the modern order.
I've seen a clip on Top Gear where they drove some early cars and the controls were completely different like levers for acceleration and braking.
How long would it take someone familiar with this 108 year old Cadillac to learn how to drive a modern car from point A to point B? 1 minute?
... speaking as someone who was in the job of enterprise BI where data sources were primarily Progress databases on AS400, back from around 2000-2010, when we finally migrated to a Linux / PostgreSQL stack.
I still have this memory that IBM sent out an Engineering Change for the AS/400 that consisted of a twist tie to help customers who had purchased an Ethernet card for it coil their Ethernet cables more reliably. (The twist tie of course then having a specific IBM part number.) I would love to be able to substantiate this memory.
IBM was also seemingly very open to supporting new technologies, both hardware and software, on the AS/400, including some that were invented decades after the machine was introduced. Usually for a fee, of course!
I don't know if 9119-590 is compatible with AS/400 though.
I saw things like that from IBM, just from their later Power AIX workstations.
For mission-critical computing companies, there was lots of process intended not to have anything slip through the cracks.
The first memory that comes to mind was actually DEC (or was it HP?), who, to send a single sheet of paper (e.g., for a license key), would routinely use an entire shipping box. Paper would be shrink-wrapped or poly-bagged, with a sheet of cardboard. Plus a packing list, itemizing not only that one sheet of paper, but also itemizing all the shipping supplies to be used.
Not very efficient in some sense, but if it avoided a single mission-critical incident for a customer, I suppose it was worthwhile.
Chances are if you bank in a community or regional bank, the core banking software is run on Power hardware. Long live the AS/400!
A TMM/TME is a hybrid role support role consisting of functions of a Dev Advocate, PM, and PMM.
While PMM would own non-technical GTM and enablement, the TMM/TME would own technical GTM and enablement.
It's an older role common in 90s era companies like Cisco, Palo Alto Networks, etc though is has been renamed as Technical Product Management now in most orgs (eg. Lacework, Netskope, etc)
Whatever tech expertise Nadella has, his EQ and ability to navigate the toxic and vicious politics of Microsoft is his superpower. They aren’t as in your face hostile as Oracle, but they are cutthroat.
Note that my take is biased, I've been a "consultant" for most of my career which is a glorified temp, and you end up in projects and organizations that hires temps. I tried an old fashioned product company once, it wasn't for me (nothing in common with my colleagues who were 20-30 years my senior, and they and the company were happy just plodding along until retirement)
How else would you set up a business that's robust to people moving on to new things?
Contrasting that with my experience I can name one company in 30 years that did that. As is so often the case in our society, everybody is doing it backwards and wondering why everything is broken.
But everyone will bullshit you to stay as long as possible with little salary rise.
But with current market it is better to stay for longer if you have "okay" project.
Also, the demand for skills has been silly for the past 20-odd years, so there’s less incentive stick around, other than money. Loyalty rarely pays off, unless you’re talking shares.
But this is rather rare and most companies have a soft ceiling for growth internally. At Google for example, for years they have been filling most Director positions externally, and so most employees find it very hard to get there and progress past that. Progression is also often subject to norms that make the sheer number of promotions required to make it high up in the company impossible.
But I decided to get out of Dodge and interviewed at Microsoft, and I asked everyone the same question. The responses were shocking to me: five years, ten years, 12 years, 22 years. One person even told me that he hadn't been with the company very long: only 18 years. And he wasn't being coy.
I've been at Microsoft for more than ten years, and I still feel like the new guy. I started at the tail end of the Ballmer days, and I'm sure it was a real grind back then, but I'm glad to see a company that -- in my experience -- treats people well enough that they'll stick around.
As the Android Runtime later demonstrated, nothing prevents you from distributing an architecture-independent representation, AoT-compiling to native code at installation time, instrumenting the native code, and then re-optimizing at runtime and/or in a background batch process if your instrumentation statistics deviate substantially from what was available the last time you re-optimized.
Apart from the extra disk space for both bytecode and native representations, something like TIMI allows for the best of both worlds as far as AoT and JIT. (It's also nice that TIMI doesn't require a garbage collector to be running.)
I'm not aware of modern AS/400 dynamically re-optimizing binaries, but doing so wouldn't break anything. Given the conservative nature of mainframe users, I imagine dynamic re-compilation would need to be an opt-in feature.
GraalVM supports AOT compilation – https://www.graalvm.org/latest/reference-manual/native-image...
But, I think "TIMI" has significantly less value today than it once did. Due to PASE (the AIX compatibility environment), there is ever more code on contemporary IBM i systems which runs outside of the MI bytecode layer. Shifting IBM i to something other than POWER would require recompiling all that code from source.
And then ARM also has CHERI / Morello, which is still at the research prototype stage, but may be production ready in a few more years
https://marketplace.visualstudio.com/items?itemName=HalcyonT...
Well, not exactly. IBMi is still widely used, as an "affordable, low cost mainframe".
I think it's a very millennial thing but I've barely gone past 3 years somewhere. Half our careers our salaries were controlled by 2008 and you had to job hop to get more money.
On my last day there before returning to Canada in August 1993, I vividly remember that Bill Gates had a general meeting where he was talking about how IBM was doing poorly and that he was thinking of buying it, but not the whole company, just the AS/400 division (obviously, that didn't happen).
Er - no.
My first desktop LCD was a Sony and did not height adjust.
Some people had furniture that raised the monitor up. Other people had the monitor stacked on top of the computer itself, and the "stack of books" thing was around.
No gender diversity either, though clearly not all middle aged white dudes.
His daughter seeing this and wondering who that was, those things hurt.
in the 1980ties my system "of choice" was an atari st and my schools computer was a VAX 11/730 running VMS 4.x ... later 2 of them in a DECnet - had some fun with them :))
but a friend of mine was an admin for an IBM AS400 and out of curiosity i visited him from time to time and "looked him over his shoulder" as we say in german during updates and similar tasks ...
was a nice and very stable system back then ...
it had a small hdd - can't remeber its size - and the updates came on floppies with 8 inches if i remember it correctly ... 4 or 5 of them at a time - around 120 k each ... laughable by today's standards, but afaik pretty common for a small to medium sized business back then - a travel agency.
cheers v.
I started balding early 20s but some of my friends in late teens.