The mainframe is alive and thriving
zdnet.com
zdnet.com
- 45 of the top 50 banks
- 8 of the top 10 insurers
- 8 of the top 10 telcos
- 7 of the top 10 retailers
- 4 of the top 5 airlines
All had significant operations prior to the mid 1990's and so have extensive mainframe investments that are very difficult and costly to migrate from. I know. I work for one of the top electrical utilities in the United States and at the heart of our customer operations lies an IBM mainframe. We're in the midst of launching a multi-year, billion dollar program to replace that mainframe. We've been talking about doing this for years and are just now pulling the trigger on it. There's still time to tell whether we actually go through with it or not, or get $100 million into the project and realize we're looking at a $1.5 billion - $2 billion effort and abandon the project.
I would not call this state of affairs thriving. Is the mainframe alive? Yes. Is it thriving? No. Most CIOs are looking for a means for retiring those assets.
One of my older greybeard coworkers tells stories of how sometimes an IBM employee would show up at 6 AM to replace a water pump (or some other component) that had phoned home the night before to report that it was beginning to fail. The hardware is super reliable and resilient but also heavily instrumented to detect & resolve failures before they caused problems. And they would send their techs out to pre-empt those problems before the customer ever noticed anything.
Not that it would have been too much a problem if it had failed - because there was so much hardware redundancy built into those machines.
One of his other stories, is a tech demo that IBM did for his company, where they put a mainframe in the back of a trailer and would drive it around to customer sites. The demo consisted of starting a huge computation on the machine, and pulling up a screen that showed the available hardware resources, then a tech would reach into the machine and start pulling out CPUs and RAM, and you'd watch live as it detected that the hardware had disappeared and shuffled the jobs around to other hardware resources without crashing. And then the tech would start swapping hardware back in, and it would add the resources back into the pool and start doing work with them once again.
All handled by the hardware and the OS.
IIRC, that's still true. I think IBM mainframes are typically leased, and the customer is billed on usage.
There's some complexity to this, where I think the metering only applies to programs in the "traditional" mainframe environment, while things running on the mainframe Linux environment are exempt from metered billing.
One of the interesting features I heard when they launched z15 was that, if one needed to restart a partition, it could tap unlicensed CPUs to start faster and then go back to its licensed capacity.
On a hot day they had an air condition problem in the data center and had to turn off a lot of machines. They tried to turn off the mainframe using the big button, but it just kept running. Then they decided to pull the power plug. System still kept running. It turned out it had an built-in UPS. After some more attempts they finally managed to turn it off.
But at what cost? Instead of getting suppliers to out-bid one another you are now stuck in a single supplier situation. [0] Building the redundancy in software allows you to run on commodity hardware that can be purchased from anyone.
Single-system-image clustering delivers on that promise with no dependence on proprietary platforms. Linux is now slowly getting the underlying features needed to enable this (e.g. process migration and checkpointing, namespacing of all system resources), largely as a side-effect of earlier developments such as cgroups. You can already experiment with Linux-CRIU to see what the basic tech is like, it's not fully mainlined yet but it's slowly getting there.
Batch workloads were things like billing: you're a utility and you need to do the accounting and billing for the million customers you have. You're MCI and you've spent a year advertising Friends and Family discounts and now need to do custom billing every month that figures out all those intersections of friends & family and sends custom billing. Not rocket science, just a lot of boring data crunching you have to do over and over again.
Interactive was everything you do today on personal systems: stores had terminal displays that hooked to mainframes for inventory control so salespeople could put together an order or cash registers would transmit sales near real time to a central mainframe and pump into/out of your ERP system.
Depending on your company's view of the mainframe world, you might have an MVS system doing all of the batch processing and a VM/TSO system providing interactive. And the MVS system could run under the VM system as a guest O/S, or you could partition the single mainframe to have MVS running on some CECs and VM running on others (e.g. you might have more interactive users during the daytime, then tune down the systems so that the MVS systems had more capacity overnight for batch operations).
Mainframes systems could be clustered together in a single data center and you could move workloads back and forth across systems while the workload was in progress. Eventually (circa 1994) you could link mainframe complexes together across DCs to provide redundancy (if a failure occurred in the mainframe equivalent of an AWS AZ the workload could, theoretically, transfer to another AZ with minimal interruption or reconfiguration).
On the interactive side…you could have thousands of systems connected to a single mainframe via various forms of I/O systems. All of your retail stores, all of the printers at those stores, frequently all of the security systems, would be set up using APPC or SNA to connect to your regional mainframe over one or more leased POTS lines.
The 3270 protocol that drove those green screens was screen oriented, if you changed two fields on the screen and hit ENTER, only the changed data from the fields would be sent, so having a bunch of terminals share a whopping 33kbps line wasn't outrageous.
From a programming and systems management perspective, the systems were designed to run as close to maximum utilization as possible. You just spent 10-15-20 million setting up this complex, you want to run it at 90-95% utilization 24 hours a day. So there's a whole set of workload management tools that developed by IBM, CA, and others to tweak applications and the operating environment to run it as hot as possible. There's all sorts of accounting built into the operating system(s) and language environments, going back to when you set literal budgets for how much a given department or program could spend.
So, this is the environment that ran from effectively the 1950s through the 1980s into the 1990s. The S/360 line launched in the early 1960s but offered some backwards compatibility to some IBM systems of the 1950s.
COBOL gets tagged as the most popular mainframe language but PL/1 (or PL/I) was frequently used (IBM had an internal variant called PL/X which was mostly used for systems software). C was used a lot as well inside IBM but usually for smaller products running inside VM/CMS.
I tried writing a web server for MVS in my spare time circa 1995–1996 and even though I was working with people inside MVS development we had a hard time getting it to work with what turned out to be a pretty crappy TCP/IP stack. The batch oriented nature of MVS wasn't the problem, IIRC the TCP/IP stack just could not handle the set up/tear down of the typical HTTP/0.9 and 1.0 connections. When a web server was eventually developed for MVS it was under the posix flavored "OpenEdition".
Net: It's just a different model of computing, with a different character set (EBCDIC, still) and languages. It's weird to see the industry kind of / sort of return to the classic shared computing resource model of the cloud with containerized applications and services. IBM in the 1990s was just as convinced at the corporate level that mainframes were dead as the consensus public opinion was at that time, which in retrospect is a shame since IBM chose to fire 1000s of people with the skills it would have needed to leverage to build a competitive cloud offering.
So imagine an environment where you can't buy your hardware, you can't buy your software, and when you've made an upgrade to your hardware your software gets even more expensive. That's a mainframe.
Our mainframe, which is mostly used for customer billing (electrical utility billing is slightly complex due to all the different tariffs, localities and taxes), and customer relationship & service management, is costing us $15 million - $20 million per year. On the one hand you'd say holy smokes that's expensive! But on the other you could say that's only $3-$4 per customer per year. Thing is you could do the same amount of processing in AWS for under $250,000 per month or $3 million per year, making the mainframe at least 5-7 times more expensive than AWS.
So how does saving $17 million per year justify spending $1 billion?
Is that worth $1 billion? That's above my pay grade and I'm glad I don't have to make that decision. For the record, our CFO doesn't think so - at least not until he's closer to retirement so if he gets canned for it he'll be alright! He's in his mid 50's now so we may have to wait another five years!
The reason given is it's getting difficult to find COBOL programmers and the platform isn't as agile as other platforms. I have to admit that IBM is working to make the platform more agile and you can do more than run COBOL - you can run Java and Linux on Z. I've been in IBM training sessions where I've been running Linux on Z and then running Docker on top of that and unless you had told me otherwise the experience was normal: you're running bash, running Docker and everything runs as you'd expect.
BUT - the problem is the core of our system is written in COBOL. There's still CICS, JCL, Copybooks and DB/2 and those who can work with those technologies are now within 10 years of retiring and the new generation doesn't know them or have much of a desire to learn them. Heck, I'm part of the first generation of PC developers from the mid-1980's and I've never done any mainframe development!
The thinking then is we'd better jump off before all these people retire and you end up having to pay $300+/hr for resources capable of working with these legacy technologies. Pay now and buy yourself a new technology set and new capabilities or pay later and just keep the system running.
On one hand, I'd say that this is true. But on the other, I'm not so sure.
15 years ago, I interviewed for a job that turned out to be for building software on OpenVMS running on Alpha.
I had honestly never heard of either thing.
The interviewer, who would be my manager, said that isn't a problem, it's just a computer that works a little different than others. Do you want to learn it? I said ok.
Over the years, the hiring process changed, and there is no way that someone like me would even be sitting at that interview where they could ask if I wanted to learn.
So companies can go ahead and spend all that money on legacy consultants and all that money on migrating their systems.
I really think big iron's customer's problems are that they don't want to pay for quality engineers, but for some reason will pay those higher rates or more for some overpriced consultants. Literally just accepting that the outsourcing dream of 2000 was not a great move and using that money on internal engineering would go soooo far. You see even .gov getting the hint with 18F and the US Digital Service building and supporting non contractor government engineers.
This is the key issue. You can migrate it off the mainframe or you can migrate it to sexier tech on the mainframe.
> and the new generation doesn't know them or have much of a desire to learn them.
One of the barriers is that these companies are not willing to pay the same money we (I think we are from the same generation - I was paid to write Apple II software) can get from working with anything that's not COBOL.
I'd love to spend some time with mainframes - they are the fastest Linux machines you can get - but don't want it enough to combine COBOL and JCL with a significant pay cut.
If banks routinely started paying Google salaries for good COBOL developers, you'd see those shortages disappear in a big hurry.
It's not pricing. As a consumer, it always seemed like the largest banks tended to be relatively non-aggressive on their rates. Isn't "we have economies of scale and pass the savings to you" the classic competitive play?
It's barely, if at all, public facing apps/websites. Okay, you've all got a branded app, you've all started offering Zelle, and you all let me take a photo of the (average of) five paper cheques I recieve in a year to deposit them.
It feels like for a while it was the vague promise of "we've got the largest network so you don't need to worry about finding a machine." A concern largely mooted by smaller banks refunding out-of-network fees and credit union networks.
Their competitive advantage is not in their software at all. It's in their relationships. The software helps them scale to large numbers of depositors. That's all.
As a younger person (graduated 2012) who entered the mainframe space for my first job then left after 2y, I can very much vouch that compensation for software devs working on zOS systems writing COBOL or HLASM is peanuts. You'd be looking at 1.5-2x'ing your income even if you took a job at a series A startup writing CRUD apps in Rails/etc.
No top 50 CS program teaches these tech. No unicorn uses them. Not a single FAANG or FANNG+ if you include other large players.
Finally, no engineering-centric company uses these tech. That's the difference between learning Go vs COBOL.
There are only a couple of FAANG+, with crazy hiring practices, and most businesses aren't engineering centric, the world is bigger than that.
It sets the trends and build what's next. RandomBigCo just follows.
Why don't people switch, then?
The mainframe gives you a predictible runtime. Long running jobs don't get rescheduled or hogged by other loads.
I know about a firm that uses a mainframe to control a factory line.
The problem with replacing stuff that works (and the mainframe does) is then your job is on the line if the transition fails, and there are a lot of business moving parts inside.
Cobol programming is a skill that is trivially taught. Any student with a Cmpt. Science degree from a credible institution, given a curriculum, can learn any programming language in a month with a bit of focus/appropriate curriculum. Can pick up the necessary frameworks in a reasonable period of time. Then it's just a matter of teaching them the line of business, and the practices and procedures.
I would claim that, given twelve months, you could teach a cohort of recent Cmpt. Science graduates all the skills they needed to be successful in any Insurance/Banking/Finance programing environment.
And these are developers that you can then pay a very reasonable salary - say, $150K/year - which is $75/hour, not $300/hr, at the cost of 1 year of salary as they come on board, come up to speed (say at an out-of-college salary of $100K/year.) - You need 1,000 of those developers - that's an up-front cost of $100mm, at which point you have your developers.
Compared to a multi-billion dollar transition cost off of a mainframe, I know which one I would chose if I were the PHB.
And this used to be very common. My first job out of school in the early 1990s was with a "big consulting firm" they had a boot camp where they taught COBOL to batches of new hires. You did that for a month, and then were deployed to a client site writing COBOL about 50 hours a week.
I've done some work in the financial space. It's interesting to compare, for example, expressing a credit card transaction in an ISO 8583 message versus a modern JSON/REST API format. The same data might be there, but one of them is going to be much harder to debug without a lot of specialized tooling.
Bootcamps saturating the market with webdevs armed with skills that barely last a year, when they could be training people with skills that would see them employed for a whole career.
It has nothing to do with bootcamps. If the mainframe knowledge was in sufficient demand, then they would be selling mainframe training too since people would be wanting to buy it.
A large part of that is probably due to American management practices, where the preference is to skimp on maintenance until they find themselves in a crisis. They don't want to pay for what they need, they want to pay for whatever is cheap.
Maybe, but it may also be true that people simply aren't aware that mainframes exist or these jobs exist.
This is not surprising. IBM could fix this overnight if it wanted to. How? By providing no-cost educational licenses for its mainframe software for use on Hercules. Why don't they do this? I believe that there are 2 reasons why: (1) they're terrified of people running their production mainframe workloads on a non-mainframe, and (2) I believe that some of IBM's existing mainframe customers keep pressure on IBM to prevent them from opening up the 'mainframe secrets to the world'. The second point is the belief in security through obscurity. I don't have any evidence to back up this belief, it's just my theory.
The other thing that IBM could do is provide low-cost (or even no-cost) options to real mainframe environment over the internet. I know that they have some of this, but it's too restrictive. I'm talking about opening it up like AWS did with free for 1 year. But that's expensive? It wasn't free for AWS to do it, and it wouldn't be free for IBM to do it. However, it would eliminate the biggest barrier to entry for new talent.
Getting back to my theory of security through obscurity, I believe that this is also in play with AIX and iSeries (AS/400). A number of years back, I bought a used RS/6000 workstation on eBay for the purpose of learning and I contacted IBM to get a licensed copy of AIX 4.3. It was as if I were speaking in tongues trying to get through to them.
IBM: "Sir, what's your IBM customer number?" Me: "Umm, I don't have one. I'm just an individual who bought a used RS/6000 on eBay." IBM: "We can't sell you an AIX license without a customer number." Me: "Ok, well give me one." IBM: "What's the name of the company?" Me: "There is no company. It's just me as an individual who wants to learn about the technology." IBM: "We have to have a company name." Me: "Put my name as the company name." ... ... ... Eventually, get assigned an IBM customer number and did finally get AIX CD-ROMs from IBM.
I can only imagine that it would be 10x worse to do something similar with their iSeries (AS/400). When you see this over and over, I'm left with the conclusion that IBM doesn't want individuals getting their hands on their hardware or their software.
You are absolutely right about how hard it is to do business with IBM. I worked for IBM for nearly 7 years, running a small datacenter. It was such a pain in the ass every time I needed to order something. IBM sales and support process is very high touch, and is designed for the large enterprise that has a team of IBM sales people supporting them. If you are that customer it is not a big deal because you have someone to navigate the process for you. If you are a smaller customer, they want you to go through a VAR. Which is still a pain in the ass.
Windows NT and the AS/400 are about the same age. This pain in the ass sales process is a big part of why Windows NT won. The IBM sales people didn't want to sell it because you didn't need to buy services with it and it was too hard to buy on your own. Where as if you wanted Windows NT, you just bought a copy of Windows NT and installed it on an easily available PC.
We won't even talk about the time I tried to use IBM Smart Cloud Enterprise.
What gives me pause for concern is the JEE/WebLogic platform is now as old as COBOL was in the 1980's when we were calling it "dead" and we're still looking at 5-7 years of development and migration time to get to the new system. What will we think of JEE/WebLogic in 2030? My biggest concern is they're going to spend all this money to get onto another soon-to-be-legacy platform that they're going to realize on day 1 they need to start planning on getting away from. This is why mainframes have been so "sticky."
Off-topic but do you work for Eversource? An ex-coworker of mine found a bug that he immediately recognized as a COBOL bug while working with Eversource CT's EDI
So, you're looking at a high risk, multi-year, multi-billion dollar project to remove your own mainframe dependency. And the business isn't fully committed to it.
> Most CIOs are looking for a means for retiring those assets.
And finding extremely expensive, high risk, long duration software projects are the usual alternative, per your own experience. Why waste prestige and budget on that when you can keep all the critical LOB software running exactly as it is (or faster) forever? Mainframes are the OG backward compatibility and virtualization platforms.
Given that, why do you think the mainframe business isn't thriving, especially given the billions in revenue that the companies you list in your post undoubtedly bring in for IBM? It's not sexy, and they're probably not winning new customers. But they're making a shit ton of money on a long lived, successful product and continuing to fund research and development on the platform.
Hum... Probably (and that was the case at my workplace) because business as usual has higher risks and ongoing costs than the migration. It's not like mainframes are cheap to own, or guaranteed to stay in existence in the future.
> why do you think the mainframe business isn't thriving
"Thriving" on my lexical implies on growing or at least sustainable. A market where new players don't want to buy at any cost and old players are all trying to get out is not this.
IBM offers a compelling option:
- yes, your software will continue to run, because that's what the mainframe is for
- yes, it will scale with whatever transaction volume you need in the future, because that's what the mainframe is for
- yes, you can get 99.999999999999999% uptime, because you have reliability built into the platform in astounding depth
- yes, we'll happily help develop new functionality and support your in house staff in managing this system; the mainframe can actually detect faults and arrange a service call.
It's expensive, but it's a safe bet in at very least the medium term. The alternative is quite risky in most cases, and perhaps less expensive over the medium term if it works out.
Nothing is sustainable forever, but that sounds like a pretty hopeful story for IBM over the medium term. Long term, who knows, but eventually we're all dead anyway.
Maybe if you are using zos without any niche configuration. IBM has discontinued many mainframe platforms in the past, and there's no guarantee that next year they won't approach you with a 10x markup on your renting costs, that will become 50x in a couple of years and will keep going up until you decide to stop paying.
> yes, it will scale with whatever transaction volume you need in the future, because that's what the mainframe is for
Only if you gave-up on data dependency from the beginning. Transactions here means independent data processing, with possible rollbacks, not the relational concept people got used to on databases. If one needs relational data, it won't scale. NoSQL can scale with non-relational data too, it's not an exclusivity of mainframes.
> yes, you can get 99.999999999999999% uptime, because you have reliability built into the platform in astounding depth
That one is ridiculous. Uptime is limited by business code bugs, network and power outages, integration problems, and well, all that causes 90% of the downtime on a single-server shop. All the mainframe promises is that the hardware won't be a problem.
> yes, we'll happily help develop new functionality and support your in house staff in managing this system; the mainframe can actually detect faults and arrange a service call.
Yes, mainframes come with support contracts. You can get support for other platforms too, with the advantage that you won't need to hire IBM.
Anyway, none of that changes the fact that nobody wants to start using a mainframe nowadays, and that the people using them are either looking on how to replace them or avoiding increasing their usage. IBM will certainly be able to milk that market over the medium term, even more if they keep increasing their prices. They revenue may even go up for a while. But that's not thriving.
Imagine you are a CIO. You have mainframe apps. Everything I said is true of your existing solution. If you keep it as is, it works just fine for the foreseeable future. Every other way forward requires a total rewrite of systems that have been stable for years and are typically mission critical financial and transaction processing systems.
That's what I mean by IBM having a compelling offering.
There are advantages to the mainframe, but there are disadvantages too. As a CIO your job is to consider both.
My understanding is at least Java is well supported on mainframes. It's not the latest greatest language, but it sure beats ALGOL.
Then Algol, COBOL, RPG, NEWP, PL/I.
Even if you say "March 14, 2021 is the grand final feature freeze for the Mainframe, and day one of the Cloud Initiative", it's not going to happen. You're not going to have a MVP for several years, and a full replacement for five million undocumented lines of COBOL for even longer. Production fires will have to be fought. Regulators will demand you change something, and won't be keen to hear "we're supporting it in the Cloud Initiative, which customers will see in 2025." So you're gonna be writing quite a few more cheques to IBM and your team of COBOL jockeys.
After 5 or 6 years, as you've unloaded most of the customers to your painstakingly built replacement, you can finally begin to scale down the mainframe staff and perhaps return/downgrade some of the rented hardware.
Given that, it may be less a put-your-own-head-in-the-guillotine move just to spend those years hiring in a few new graduates with the promise of "We'll apprentice you to handle our unique and obscure codebase, and as long as you show up 40 hours a week and don't take a dump on your manager's desk, you're going to have a reliable paycheque for the next 20 years."
Which ones has it discontinued? I thought once the 360 platform got going, it was basically forever (unbroken chain of compatibility). What am I missing?
That might make it good business for a long time but it would be a stretch to call it thriving, and it provides gentle but salient cause for businesses to consider migration.
I work at a company that basically built its current business on mainframes in the 70s/80s/90s. The migration effort to get off them has been going on for more than 20 years and still isn't done yet.
Yep. They're already spinning up a big cloud migration project that will eventually replace many of the re-engineered on-premise systems, and they're still not yet off the mainframe (they're close, but that last 20% is always the hardest).
I've been hearing "mainframes are dead" since the 1980s. The market for them has dramatically changed, but by the same token there's just more people in IT who have no clue what the mainframes do from a business perspective, so "obviously" it's a dead end.
> And, since we're keeping score, zero content snark is bottom of the heap.
That's just hurtful, man.
Datacenter-scale compute on non-proprietary platforms (e.g. leveraging OCP standards and best practices, which have been developed w/ significant input from FAANG and 'cloud' vendors) can easily tackle any mainframe workload. The implied path is obvious.
That being said, it takes a massive commitment and pretty much a rewrite from scratch to achieve a jump from mainframe to modern DC. You need to embrace individual node failure and that's not something you can just hack in.
AWS, Azure and GCP are all proprietary, and all are producing larger and larger servers and VMs. They are basically rentable, virtualised mainframes at this point
Most of us didn't consider writing cross-platform software back then and the few who did were writing software for Unix, not Windows, and that because there wasn't a single dominant OS (Unix was a fuzzy target) and binary architecture you could write for, so you had to think about portability.
Oddly enough, there was a lot of cross-platform work on games in the 80's because the home computer market was very fragmented.
For graduates of my (very small) high school, Classics was the number one choice of college major and though I'm not in close touch with all of them I never heard any of them claim that they had made a mistake.
Cobol is sort of like Lisp* in that regard: the options are quite limited, but as the supply is too you'll probably do better than fine financially.
* wow is it painful to write that.
Which is why I used the vatican comparison.
It is also relatively rare for new languages cite Cobol as an inspiration for any of their features.
Cobol has a feature whereby dissimilar structures can be assigned. The structure members which have the same names are identified between the structures and the data is moved accordingly. Or something like that. A programming language which has such a thing could claim to have had that inspired by Cobol.
If I ever add that to TXR Lisp, I will be sure to drop a Cobol reference into the documentation.
I might have learned about this via Knuth; I think there is a mention of it in some passage in TAOCP.
This horrifies me on several levels, though I suppose it could be considered a sort "cast to a superclass".
Indeed, see Volume 1, section 2.4 (Multilinked Structures). Page 426 has the example of the statement “MOVE CORRESPONDING DATE OF SALES TO DATE OF PURCHASES”, which, if I understand correctly, means something like “for each field foo in Purchases.Date, copy Purchases.Date.foo into Sales.Date.foo”.
The discussion is in the context of what kinds of data structures would be needed to compile such statements, which I guess makes sense considering that the original publisher offer for TAOCP was to write about everything a programmer would need to know to write a (60s-era) compiler.
(Coincidentally: I've started attending an online meetup that someone is running for reading TAOCP—we usually read about four pages in the two hours—and this is the very topic that we're going to read tomorrow.)
Are Latin and Ancient Greek dead? Ostensibly yes. But not useless. Besides teaching, there is a tremendous amount of books and texts written in Latin and Ancient Greek. To study them, one would invariably need to learn the language.
Texts from the ancient world are handed down to us not from on high, or a single authoritative source (please consider the problem of duplication prior to computers or mimeographs), so considerable effort and erudition is entailed in comparing sources, filling in lacunae (here's a true warrant for what most consider a sesquipedelian word), all of which is predicated on perusing (in vain) an intuition for "pure et clare latine" and a hard-nosed insistence on _attested sources_ on matters of usage and expression.
I treasure, uncountably, the continuity afforded by training new generations in "antique" languages and dialects. It addresses a much deeper and more valuable problem than shitting out the next side-hustling platform.
These days, most companies seem increasingly resigned/accepting of the fact that they're not going to find young, dynamic COBOL programmers in the wild. On top of that, old-school COBOL has a knack for creating idiosyncratic software architectures. Learning how some particular monstrosity was taped together is more important than the language itself, which is brain-dead simple. And there are few such COBOL codebases out there in the wild to read.
The result is a move towards hiring and training good programmers in-house, quite counter to the general trend in the industry. Which makes for good job security but also makes it hard to enter directly.
If I could go back 30 years, I’d have made more money, slept better and would have more hair on Z. Not that I can say for a new generation but now that everyone is a “full stack” developer and the stack changes every few years, I’d still prefer longer term niches if I was 22 again.
Compared to webdev? I'm very skeptical. The following scenario seems more likely:
I’m not savvy on the costs, but I think the main benefit of ditching the mainframe is you can start using the IT tools, expertise, workflows, languages, frameworks, best practices etc. that 99% of the world is using.
With the mainframe, you’re in sort of a shielded of nook of the software world, where everything is fast and stable but also juuust a little different.
Key theme is slimming down the mainframe to bare basics, then take out either batch, or DB2, and then see to pull the trigger on the rest, 6y horizon...
But if you are spending a billion dollars, how much code do you have? Also, on the lack of programmers, just pay them more and you'll get a lot more applicants.
May I ask Why? If they are working perfectly well with a decent roadmap and upgrade path and quality assurance from IBM?
[0] https://www.cnbc.com/2020/04/06/new-jersey-seeks-cobol-progr...
That’s when I got out. When it came to switching jobs I had zero experience in current tech and had to take an entry level position and a pay cut. A few years later that same company had massive layoffs and all development outsourced by 2008 or so.
So yeah whenever you hear about opportunities in mainframe cobol don’t believe it. I’ve got 2,000 former cobol developers from that company alone who will attest that its farce. What puzzles me is who’s behind this idea and what’s their goal in promoting it?
I'm surprised to read these recent anecdotes (going back to that New Jersey unemployment COVID programmer shortage) from various people that COBOL jobs are in high demand yet shockingly underpaid. That would seem to imply markets don't actually work the way we've been told.
The difference in this case is that the press is showing sympathy to those who won't pay the price and then complain about the missing supply. That doesn't happen in most markets.
Why is the press listening? Because the people complaining are people that the press normally listens to - banks and big, big businesses. It seems that many such businesses would rather complain, publicly, than pay a market-clearing rate. Well, I don't have a lot of sympathy, to be honest...
The other way of reading the situation is: There's nobody interested in working in COBOL at the suggested rates.
a bit off topic, but this is a huge thing that I at least didn't pick up on across jobs for many years - for accounting reasons, many firms will always see what your team is doing as an 'expense' to be 'minimized', no matter what you do, and in many cases, there is nothing you, or your manager, or even your managers manager can do to change this, since it is often decided at the highest levels of the firm based on broad-stroke accounting policies that have a direct impact to how the firms entire books (and therefore revenue/profit/etc) are calculated.
IT/tech related? goes in a cost center.
What's a cost center? an optional expense which can and should be minimized, with manager compensation often tied to how effectively costs are controlled.
If anyone working in tech is not aware of the various accounting models used in business, it is well worth your time to get familiar with the basics - assignment of 'cost center' to technology groups in firms that use cost/revenue center models has a huge impact on how teams are likely run, decisions are made, and potential for career advancement for employees in those groups.
https://en.wikipedia.org/wiki/Cost_centre_(business)
https://en.wikipedia.org/wiki/Profit_center
this is not the only costing model (job costing, activity based costing, etc) , but one in my view that is often hugely detrimental to technology teams in non-tech companies unfortunate enough to find themselves pre-categorised as a 'cost center'.
Think this is highly dependent on the school/student; In my experience (data point size=1), MBA teaching hasn't really caught up to the fact that basically everything is technology these days, so it shouldn't be treated as something 'separate' to the 'real' business, which it often still is - i'd wager that the resulting compartmentalization in thinking around how to manage tech is the main driver behind allowing poor systems to capture IT value from the financial perspective...
1) the Lindy Effect: https://en.wikipedia.org/wiki/Lindy_effect
2) The mainframe's reliability-centric culture vs. today's flaky software design culture
Very underrated comment! Ran into the same thing with VOIP vs. traditional PBX systems. The IP Telephony guys were sloppy with uptime and reliability, and that's being generous :p
I have to respect the reliability aspect of those old systems they were built to last a generation.
Seriously. if you pay IBM for a couple of z series machines, you can link them together in step, and literally blow one up, and you'll only loose a certain number of CPU cycles.
sure they are expensive, but its cheaper than buying a team of ten engineers for x years to make K8s do the same thing.
Also, you don’t buy mainframes, you rent mainframe CPU cycles. Or actually, you will pay for the rolling window 4 hour peak of CPU utilization.
IBM will gladly ship out the fridge sized box that holds those CPU’s to your data center for free, as soon as you turn it on their cash starts coming in.
By that time there were a few other mainframe class manufacturers, most notably Tandem and Amdahl.
Notably missing (and most relevant to the 2020s tech scene) was DEC, who by the late 60s was making smaller mainframes as well as minicomputers. What makes them different was their decision to tae a different, and what Clay Christainsen would call disruptive path. And while highly influential they were themselves nuked by a new class of disruptive companies.
IBM has reinvented itself several times over in its almost 150 year history. Not sure the current one is successful, but I suppose it's still too soon to tell.
Let's say both are running the same OS and application using VMs, and neither is doit specialized types of compute (i.e GPU stuff), just processing gobs of transactions.
Obviously there are big internal architectural differences, but ultimately these must manifest in some quantitative and qualitative differences and tradeoffs from a black box perspective.
Please discuss.
From the buying manager POV, one is cheap commodity and the other is expensive single vendor equipment.
From the programmer POV, one is a proprietary environment with uniformly addressable memory (not UMA, just addresses) and the other is a distributed blank canvas where you must write the IPC yourself (or use a library, of course). Both have a huge IO capacity, but mainframes can integrate data much faster (and usually, process it much slower).
From the ops POV, one is a single machine (a heavy one), and the other is an entire datacenter.
It's a single server with massive amounts of cache (like 1Gbyte or more of the L3 cache) and optimized everything.
When you login via ssh to your server, each keypress creates an interrupt which is handled by the CPU. If you are using emacs , even on the terminal it might be multiple interrupts due to screen refresh etc.
Mainframe: generate the form or the screen data. Send it. 1 interrupt. Your terminal or terminal program handles everything until you send 1 or more lines (not characters) back. A dedicated processor handles communication with the terminal. It then bundles up whatever the changes are and sends it to the CPU. 1 interrupt.
Disk, network etc. is the same, with dedicated processors on the network card or disk controller that go out and send/retrieve the data without CPU intervention. Database table retrieval: the table is sent to the CPU, already sorted in the requested order. Etc etc.
Now that is crazy cool.
Two years back Db2 has become strategical database previously it was oracle in the decision tree. Security and up time has been solid.
Just seeing the trend at workplace on work related to mainframe, I would say it's picking up. Basically we know the work(projects), so we are focussed on onboarding process for newer applications as we are more and more working with people who have no idea about mainframe or who were once on the bandwagon saying mainframe is dying once.
Interns are trained internally after college. I would say that's the best training.... At first, they hate it. It's all black and green text, but slowly they pick up.
Is it even possible for an ordinary person to rent a VM on one of these machines and do some compute? If I have a startup that needs to crunch a bunch of models or would have an easier time running some system centrally versus having to have the skills & time to create a purely distributed system, why aren't mainframes a good/better/easier option? How come AWS doesn't have any? Does IBM have quick/easy on-ramps to hosted mainframes via Softlayer (I can't find them if they do)?
The only real remaining advantage is probably the overall reliability, redundancy, engineering, etc. Like being able to swap out a CPU while the system is running.
The chip is impressive, because they don't skimp on costs, because they sell them for a million a pop, but consider how many ryzen on xeon cores you can get for that price, with all the cool DRAM and IO tech you want.
https://en.wikichip.org/wiki/ibm/microarchitectures/z15
compare with:
https://en.wikichip.org/wiki/intel/xeon_platinum
https://en.wikichip.org/wiki/intel/microarchitectures/cascad...
A mainframe (in recent decades) is essentially a proprietary, high cost, tricked out server farm in a box (which is often big enough to be a small building, btw), which might make sense in a few cases but mostly is not as good as a much larger, non-proprietary mainframe, called a "server farm", which is supported by several competing vendors.
Well, server farms don't usually expose a single system image. You can't set the OS to "migrate" workloads seamlessly across hosts the way it can migrate them across CPU cores. Linux is now getting some of the underlying features that are needed to enable this use case, but integrating them into a coherent whole will take some time.
In a way, the kubelet is the kernel of a k8s cluster. You wouldn't run masters on every node in a cluster, because consensus gets harder quadratically. Maybe workload migration doesn't have to live in the (linux) kernel.
Hybrid cloud just means you have all the security problems of hosting your own infra, AND all the security problems of the cloud. Not to mention other non security problems of both systems.
When these CIO / CTOs make this "middle of the road" decision, they are dooming their tech budget to spend more on infrastructure and processes than actual product.
It's one thing to have a "we are both until we finish migrating to one" but to try and maintain both forever is just punting the problem to the next person.
Since they are all Turing machines perhaps we can also use AI to generate virtual COBOL programmers from disk images.
Anyone interested? ;-)
Because mainframes are more than just a quantity of computation. Their hardware and software has prioritized reliability and resiliency far more than the PC architecture, which pretty much everything else is based on nowadays (and which started out and remains mainly about low-cost).
In some ways the approach you describe would be like replacing a ZFS RAID array with an a single large disk with a FAT filesystem. It's possible, but you probably don't want to.
Full disclosure: I'm not a mainframe developer, but I've read quite a bit about them.
1. It’s difficult to obtain skilled personnel, that’s it.
Why, because you can only truly learn mainframe technologies on a mainframe. Hercules and all the things that IBM have done themselves such as MTM, VS Code extensions are not enough. The barrier to learn mainframe technologies is too high.
I agree with your main point though. Occasionally I’ll see someone recommended people to learn COBOL to get a job as if the language is the most relevant part of that industry.
There are more Debian GNU/Hurd systems than Debian s390x systems :)
https://www.theregister.com/2017/05/24/hitachi_exits_mainfra...
https://en.wikipedia.org/wiki/Amdahl_Corporation#Amdahl_cust...
Shame IBM won't sell 'em to us...
I do not know whether or not mainframes are the right solution, though. I don't really know anything about them. I don't know whether or not the architecture is the correct approach.
>IBM has grown usage as measured by MIPS, a method of calculating the raw speed used, by 350% over the past ten years.
That's an awful metric. MIPS isn't even a good way of measuring aggregate performance and aggregate performance is a terrible way of measuring adoption.
This is probably not true in industries such as banking, manufacturing, agriculture, etc. I worked as a field engineer in a food processing plant once, during the peak season for the crop being processed. The plant manager told me that downtime could cost the plant up to a quarter million USD per day.
> Why do we accept that our hardware is unrelaible and then spend so much effort making our software tolerate failure?
From what I understand, the whole point of mainframe design is to tackle the issue that hardware is unreliable. Mainframes feature many hardware redundancies, moreso than a commodity rack server.
> I do not know whether or not mainframes are the right solution, though. I don't really know anything about them. I don't know whether or not the architecture is the correct approach.
Mainframes are designed for throughput, not performance. They are good at things like transaction processing, not machine learning.