IBM Z mainframes revived by Red Hat, AI and security
searchdatacenter.techtarget.com
searchdatacenter.techtarget.com
However, the big System Z is surrounded by a lot of marketing and sales people who only know how to hype the shit out of the world. You talk to them for 2 minutes, and they bring up Watson just like in the linked article. For no logical reason. It is a big percentage of your job to cut the bull shit if you are dealing with IBM, particularly the System Z crowd. Unfortunately.
As expensive it is. It might be a crown jewel of enterprise computing even in the modern era if the ecosystem around it evolves to modern practices and cut out the bullshitters.
I have run a few thousand Linux VMs under Z/VM (hypervisor) on the Z10 hardware, using specialty Z processors made for running Linux. It was lovely albeit being inaccessible to the masses.
For example, the hypervisor on P series has been around since the introduction of Power5 and it allows you to create micro-LPARS with a small fraction of the CPU resources. Even back in the Power5 days, there was something called entitled capacity that lets you specify the minimum amount of CPU resources your LPAR would receive even if the entire server is being hammered simultaneously. It's an engineering marvel. Back then it was called APV. I believe it's called PowerVM these days. I'm sure that most people in our industry have no idea that this capability exists, let alone that it's been around since Power5. Sad, really.
This isn't really accurate WRT what is happening, the IFL (https://www.ibm.com/products/integrated-facility-for-linux) is little more than a normal fully unlocked PE that is only licensed for running linux. Its just a cheaper way to get an unlocked PE.
Long before I joined the company, that department tried to rewrite the system to be "modern", porting everything over to Java and having it run across a couple of servers. This was in the late 00's if I remember correctly. Apparently this rewritten system couldn't handle the volume of data that the mainframe can easily process during its batch cycle. However, were they to rewrite the system now (having the tools available to better facilitate real-time processing along with flexible resource scalability) I bet that a new system would be able to keep up with the demand and be just as expensive, if not a little more cost effective to run.
Does anyone else have any experience updating and rewriting old mainframe batch processes to newer systems and architectures?
I also recently came to learn that the mainframe run jobs in batches, so I'm not sure how does that affect performance, if it even does.
Also, mainframes can run both batch jobs and "online" (CICS) jobs. Our shop is about half and half with batch and online programs. The main thing that batches can be annoying with is that they run on schedules and may have to wait for other jobs to complete before they can kick off, causing delays. We just upgraded from a z13 to z15, so our mainframe is no longer a concern but we are also much smaller than a bank.
If moving off the mainframe becomes cheaper, they have a lot of flexibility
My limited experience with this is this strategy is that it works, although it is not error proof and can be messy. You also end up discovering that assumptions you made about the legacy software are not correct and then you need to back pedal and potentially throw away previous work (ironically your own assumptions about how to modernize the legacy code become their own legacy that you need to work around as the project ages).
It's good if you have some of the original developers still around and probably worthwhile to give them consulting $$ in order to bring them out of retirement.
Perhaps the best thing about this strategy is if you in the end conclude it isn't worthwhile, you will still have modernized some sections of the codebase. Of course, because of this you need to make sure that the individual section by section modernization efforts are themselves a net gain, instead of just hoping that the final end of the modernization process will be a benefit, because you might never get to that end point.
Also, you might be able to enhance the product early on by being able to iterate and improve the sections that have been modernized, instead of waiting till everything is done to flip a switch and get new features.
The downside is this is a slow and gradual process. In the meantime your developers are filling their heads with details of a unique system that is hopefully going to go away never to be seen again.
If you have specs and documentation and tests and all of that good stuff, it might be better to just start from scratch, and build things up according to requirements.
Also, at our bank, very few applications use the REST API. Since most applications are developed by contractors, most vendors don't want to use our custom API, and instead opt to connect to the mainframe directly, because then they can reuse the code at other banks after charging us to develop it.
My experience with connecting UNIX to Z/OS was not with a real API (HTTPS) but UNIX > TIBCO EMS > something called TIBCO Substation > CICS. It was horrible, mainly because of Substation which needs to translate on the fly between two completely different platforms.
IMO any "modern" architecture can handle the load. After all Google, Apple, Facebook, Microsoft, Amazon, etc. are not running on mainframes (just taking mastodon examples).
The main difference between ARM, x86 or Power vs Z is that you can keep your CPU usage between 95-100% in production and not break everything.
Performance is really not an issue, especially when you factor in the price (we could easily double our servers and still be cheaper than a pretty baseline Z13).
Main pro of the mainframe is: you can basically not touch your code for decades and it'll still run fine, every technology on it and the hardware itself is rock solid.
Main pro of leaving the mainframe is: you don't need to worry as much about expensive/hard technical debt (i.e. EBCDIC text encoding, special hardware for everything), pricing/billing is much simpler and cheaper, way easier to switch technologies.
We run a mainframe shop and for us the cost is fairly reasonable, and the high bandwidth/reliability (even with the cpu pinned to 100%) is the most critical aspect to us. Ignoring the risk and cost of migration, I am not even sure that an alternative would be cheaper... but it is quite difficult to determine the cost and my boss obviously factors in the cost/risk of migration anytime this comes up.
I'd be curios to hear of any successful total migration attempts. Most shops I know of keep their mainframe and build around it with modern tools and we do as well.
Another company I interviewed with did just what you said, they built REST APIs and other "modern" interfaces to the mainframes so that 99% of app developers in the company don't have to learn anything about the mainframe.
I've stayed in touch with some friends there all this time, and apparently they still haven't finished this project as of 2020. Disaster after disaster.
The first executive couldn't execute, so he moved on and the project was slow-rolled for awhile. Until enough time had passed for a subsequent executive to claim it as HIS "fresh new idea".
That executive came it very aggressively, actually pushing out all of the RPG employees and contractors well before the company was really ready. So that executive was pushed out, and his replacement had to hire IBM Global Services to come in and take over everything. IBM hired most of those employees and contractors who were let go. So the net result was the same people doing the same job, with the company paying 2-3x more than they had been paying.
It was around 2015 when they finally reached the tipping point, to where the company was "a Java shop with some legacy big iron" rather than "an RPG shop with some Java e-commerce pieces". But there's still a lot of AS/400 on the inventory side of things, and it will probably be another couple of decades before that finally goes away.
A lot of people on HN and Reddit, who are either students or startup employees, just have NO IDEA how things work for large businesses in the other 49 states. Stuff lasts forever, it's just a completely different world.
I am Not sure why languages are suddenly considered obsolete, I think it is that all the programmers suddenly run over to the next big thing leaving massive gaps in staff that mean recruitment becomes a problem. If I was in the market today I would be running the other way and embracing what currently works.
"Maintained a COBOL system that processes twenty zillion transactions per hour" just doesn't sparkle like "replaced 40-year-old monolith with a galaxy of JavaScript microsoervices that get winded at 100 requests per hour."
A fair bit and key is that the business process and understanding of all that is more key to getting a good transition over anything else.
The other key is small chunks, identify aspects that can be shifted across bit by bit - reports that are using daily data can be a datawharehoused data affair with your reporting system tapping that - that opens up tools for the the various departments to do ad-hoc reports by themselves without suddenly spiking the key-business loads.
But so many avenues and aspects there is no simple list to go thru, as each one depends upon the set-up and more so - business logic.
So I'd take a COBOL programmer who knows the bushiness over a top-end java programmer who does not know the business for a migration project as be easier to teach the COBOL programmer java than it would to teach the java programmer the business logic as would take sooo long and to learn all the nuances and what needs to happen when things go wrong and what is priority and the dependencies. Alas many management and project types will see a glossy top-end java programmer and pick them and ignore their in-house talent all too often.
However - batch stuff is maybe the easiest area to work upon - less users in the equation and with that - less variables to go wrong ;).
Mainframes are also not just large legacy number crunchers - the whole aspect of resilience and fault tolerance goes much deeper than ECC memory, RAID and dual NICs and PSU's. Much deeper and that is why they have up times above and beyond your X86 server class. This with database and messaging systems equally tried, tested and robust. Which gives many factors to work upon.
Though as always - business logic and process along with legal requirements and other constraints for data-flow are key and certainly flowcharting that all out and the dependencies would be key to have anyhow and essential for any migration.
The same goes for all wire transfers, credit card transactions, ATM infrastructure and so on.
The banks solution was to simply write about 40 different Javascript and Visual Basic interface apps for staff to use, which are poorly maintained failures, running inside on-premises VPS. If you wanted to access the apps for one of the banks subsidiary banks... you had to open a VPS via remote desktop INSIDE the remote desktop instance of the VPS you were already working on.
Where it got really stupid though is they wrote a windows program which emulates the 'green screen' environment. My coworkers in their 50s were using the same IBM commands to update client account details that they learned during training in the 1970s as branch staff.
>Does anyone else have any experience updating and rewriting old mainframe batch processes to newer systems and architectures?
What we did is... use AutoHotKey to record keystroke macros which are then executed into CICS via a windows emulator. It sounds stupid, but we managed to automate about 50,000 FTE (full time equivalent) of man-hours from our India based back office teams. The MBAs loved that. Of course, this was achieved by using an auto hot key script on an IBM terminal emulator running CICS inside a windows vps running inside a second windows vps, running inside a desktop computer. All written in a mixture of VBS/javascript/AHK
Ultimately, you actually often need something like an IBM to be around to acquire your small company so you can run away with the money.
I disagree. From a marketing perspective, they are a company that is known to over promise and under deliver. Their marketing is vague pie in the sky without much to show for in the market. Watch this IBM ad and tell me what is it that this company actually does:
https://www.youtube.com/watch?v=gsEwmumEsKU
They are a company that fails to innovate, and fails to be relevant, yet they put a ridiculous amount of effort to try to convince people otherwise.
From a business strategy perspective, IBM has spun off or sold off most of their great product lines and business units. Some of which were failing under IBM leadership, and became profitable once divested. Every company they acquire, they seem to dissolve after a few years.
From an internal culture perspective, they are known for laying off their talented experienced employees who have been with the company for over 20 years in favor of younger, cheaper employees. In some departments, they have resorted to simply not promoting people and let people to leave on their own accord.
That being said, most of my experience in working with IBM has been while they were under CEO Ginni Rometty who led from 2012 to 2020. This was one of the biggest bull runs in history, yet the stock went from about $205 per share down to about $120 per share. They have a new CEO now, so things might change.
They do innovate, it's just that the places where they have been putting all their flashy marketing over the past couple of years have not been the places where they've been innovating.
They have not one but two custom CPU architectures designed entirely in-house (POWER and Z) which they've kept competitive over the years (in terms of hardware if not price). They're one of a very few entities (corporate or academic) at the very leading edge of quantum computing and one of the few builders of high-end supercomputers (#2 and #3 most powerful computers in the world at the moment are POWER systems)
The problem is that while they are legitimately extremely good at many things, those things make up a small fraction of the "surface area" of IBM and it's not what they have have historically been strategically emphasizing for the past couple decades. The average engineer's experience with IBM is more likely to involve their software products and consulting services and/or sales team than it is any of those things I mentioned.
I'm legitimately curious. I already know how stupid reliable they are in terms of correctness guarantees and staying up when hardware failure and replacement occurs, but what truly sets them apart, considering their cost?
At least for we 20-something engineers, I'd wager that nearly all of us have never had an opportunity to be exposed to mainframes.
Sometimes, these systems are unable to be run in what you'd think about as a traditional distributed systems model. You don't just run that one COBOL program 100 times and kill one process everytime it jams up.
The software inside of some of these systems was written in a time when it could be optimized to the absolute bit. At that level of optimization and skill in development, you're not really going to do better or get more throughput with a rewrite. Instead, you just keep wrapping that thing in outer layers and message queues.
So, if you need a computer that can absolutely 99.99999% never stop working, including redundant everything (Power, ram, cpu, network) and you're running something that's super old but super efficient and has worked basically perfectly for 30 years, you just buy another one to replace the original, and add some containers into the mix for the queues, or storage adapters or whatever, rather than spend a billion dollars re-engineering the whole thing from scratch as microservices that may or may not work.
That, or it's a giant fucking database server. That's usually the case: one big floppy database that runs everything in the company and it's SAP system.
Another interesting feature is that they are designed to be monitored, because a traditional business model was to lease a mainframe and rent out capacity to smaller companies that'd pay for things like CPU seconds or bytes read/written.
They might also know others who do it too. I must admit I've never heard of such a thing from non-IBM sources.
[https://medium.com/selency-tech-product/your-balance-is-0-30... - all those problems are addressed.
Also, per [this thread](https://softwareengineering.stackexchange.com/questions/1129...):
> In the environments where COBOL thrives, multi-threading is usually left to application containers like CICS or IMS which are good at it, rather, than programmers who tend to be bad at it.
They don’t give any more context to know how unusual that 69% is. Maybe there is a big jump every year in the second quarter, and this one was just a bit bigger. I don’t know.
One reason they cite is the increase in online shopping that prompted customers to pay for activation of the extra hardware that is included, disabled until you pay, in each mainframe: “Some industries are in slumps, but online sales are up and that means credit card and banking systems are more active than normal. They liked the idea of being able to turn on ‘dark’ processors remotely.”
However... then you have to: 1) Deal with IBM's ridiculous sales staff. They're rarely technical and hype their random software offerings like there's no tomorrow 2) Straight up purchase these mainframes and put them in a datacenter somewhere. No easy way to just run one of these things on eg; AWS even if it's dedicated just to you. 3) IBM licensing is beyond stupid. The typical workloads people run today wouldn't be cost effective purely due to licensing. 4) The tooling is ancient and highly proprietary. People want to be able to write modern C++/Go/Rust/Java and if you want to use specialized libraries to interact with the hardware that's fine. But the current ecosystem is BAD. 5) How do you even learn to use Z/OS? If nobody can just play with it even in a free emulator then why would anyone even consider it?
If IBM bothered to address 1-5 maybe we'd actually see more people considering mainframes at larger companies. But they're digging their own grave.
All points listed are very valid BTW.
Is there any justifiable reason for a greenfield project in 2020 to choose mainframes as a platform?
Their reputation is no longer deserved imnsho.
My experience is with highly-skilled staff who are very comfortable with using the IBM troubleshooting facilities of the product, you very quickly (a few days on normal support cases, 2-6 hours on Sev 1 cases, about an hour on Sev 1 where you're paying for enhanced support) get to real developers with real source code access.
The problems come when you aren't using skilled staff. There has been a huge upturn in support cases I've see in clients' IBM Support case histories that are really "professional services/training-through-support" incidents, where it is painfully obvious the requesting parties don't know the most basic aspects of the products they're opening support cases upon. Both customers and IBM are shooting themselves in the foot here: customers for leaning on IBM with unskilled staff, IBM for accepting it. No good solutions without spending more money (on both sides), unfortunately.
Everyone who snarks about enterprise software: the road to enterprise scale dominance is paved with a maze of twisty little requirements, all alike (on the surface, then you peel it back and the cockroaches of edge cases come swarming out). There are more things in IT, Horatio, / Than are dreamt of in your philosophy.
That was a fine comment right up to the snark. I've been in IT for 35 years now, a portion of that on mainframes.
Yes, customers are often also to blame. But IBM is no longer equivalent to competent. They have plenty of incompetents on staff and do not hesitate to let those fend for themselves where customer data is on the line. A recent case involved fairly massive data loss for a customer, and I'm still surprised that never made the news.
IBM is still quality hardware, and very good documentation on that hardware but the professional services organization is a mixed bag.
For IBM professional services, you'll have no quibble from me. Very mixed bag from what customers tell me, and customers nearly have to be domain experts to make smart purchase decisions on contractors. A non-trivial chunk of big name professional services placements (not just IBM) are third parties, most of those third parties get run through mandatory sub-sub-contracting relationships and there is even a thriving industry feeding off of sub-sub-sub-contracting relationships.
IBM Support however, IME while it has declined markedly in L1 and marginally in L2 (you need to get to the L2 leads quickly if you know what you're doing and want to save time), is still quite good in L3, and if you pay for the enhanced support, exceptionally good (but it is so expensive that few people outside the IBM ecosystem have even heard it is an option).
IBM's documentation has always been very thorough, but I've seen a decline in accuracy and editing (to organize the typically voluminous standards) that corresponds with their gutting of the US technical writing teams who when I worked with Redbooks upon were typically very technically proficient in their own right. I try to help out these days by insisting upon a documentation APAR when I run across documentation defects in my accounts, but I'm likely in the minority of users. Ever since they took away the ability to comment on individual pages in Knowledge Center, they've considerably increased the friction for users to help increase the quality of documentation.
YMMV of course, just sharing my perspective.
IBM z15 Technical Introduction (3.3m PDF) http://www.redbooks.ibm.com/Redbooks.nsf/RedbookAbstracts/sg...
IBM z15 Technical Guide (22.1m PDF) http://www.redbooks.ibm.com/Redbooks.nsf/RedbookAbstracts/sg...
(Please forgive my URL handling, newbie here!)
It also says a lot how relational database models outlive business logic api-s.
https://www.openmainframeproject.org/
Also Debian runs on IBM Z mainframes for folks who prefer that to RedHat:
Last 3 years, lots of things are happening, more offloads to zIIP than General CP. In terms of applications more and more Db2 Queue replication for the time monitoring. CDC target being Kafka, IDAA for analytics and after acquisition of RHEL, now I see admins, sysprogs installing Spark, Anaconda, Jupyter in mainframe.
Usually in mainframe for detecting frauds, it used to be rule based now some companies are planning for machine learning in mainframe and be close to source and detect frauds.
Only thing mainframe cannot support is Deep learning as that requires GPU, no GPU here.
Anyones companies already started ML and spark in mainframe and what's your experience till now.
You can have racks full of PCIe slots if you want. I don't think there's anything that prevents you from adding a dozen GPUs on every IO drawer if that's what you want. And, unlike PC-based servers, there's a ton of silicon dedicated to keeping all IO channels fed, so, you'd probably have very low CPU usage.
Mainframes are just too expensive, even with open-source software running from the Canonical or Red Hat ecosystem. You just can't keep billing CPU consumption and sell memory sticks like it's made of diamonds. Running heavy "modern" workloads will give you a crazy stupid high R4HA MSU bills ... the clash between the relatively cheap commodity hardware philosophy of the last 10 years and the IBM way of billing high-end things is too big.
I would say mainframes are still really good at handling an ultra large bus with sky-high bandwidth but that's pretty much it. The power platform is a lot cheaper and also good at that.
[1] Earlier this year: IBM’s Lost Decade https://news.ycombinator.com/item?id=22224782
https://en.wikipedia.org/wiki/Service_bureau#Histories
I have no idea about the current level of activity in this sector (mainframe proceesing as a service), but would not be at all surprised to find it greatly diminished from it's heyday when dinosaurs roamed the raised floor.
I don't think it takes a huge leap to draw a connection between these businesses and the cloud providers of today.
Generally Billing is in peak MIPS.
I wonder if the math works out to buy or lease one and rent out cloud compute instances.