Hacker News Reader

Comment by idlewords | Hacker News Reader

This is a fascinating bit of nerdery. Would someone be willing to explain what mainframes are used for in a modern context, and how they compare to a really high-capacity x86 machine? I know that one difference is that you can hot-swap pretty much any component on a mainframe, but would be very curious to learn more about these beasts.

View on HN

A “really high-capacity x86 machine” is really just a machine with a lot of CPUs and memory. It is not a machine that can feed each one of those CPUs, or DMA to each one of those memories, from separate network or disk IO sources, all at once. Basically, there’s only one PCI-e bus, and it’s a bottleneck. That’s why, for IO-intensive operations like search indexing, you don’t scale x86 vertically, but rather horizontally, with map-reduce strategies: you need each core and each DIMM of memory to have a commensurate IO bandwidth available to it. And that’s also why a 64-vCPU VM on shared hosting will always underperform a 64-core dedicated instance from the same provider: with the dedicated single-tenant host machine, you’re not fighting over an “oversubscribed” PCI-e bus.

Mainframes are built differently, such that each CPU and NUMA zone of memory has its own (virtual) private bus to its own (dynamically) dedicated IO peripherals—it’s like a little circuit-switched network with CPUs “placing calls” to peripherals. And, because of this, mainframes do let you just scale your problem vertically, without IO bandwidth limitations ever getting in the way. They function less like like one computer, and more like a Mosix cluster that all happens to be running on a common backplane. But, unlike Mosix and its ilk, the DC only sees one SNMP node with one set of traps, and the whole thing provides hotswap robustness against itself rather than that robustness being partitioned per machine.

The other day I described a modern mainframe as something that resembles a cluster where CPUs are connected through very high-speed low-latency buses (to the point they can behave like a single-image NUMA machine or be partitioned into multiple smaller ones) connected to multiple networks of specialized computers each running their own tiny specialized OSs that manage IO over different buses. It's common for them to have hundreds of PCI-e buses (and cards connected to storage, communications, other mainframes - for failover or clustering - etc).

An x86 server is a computer built around its CPU. A mainframe is a computer built around its buses.

"Single system image" is a real approach to clustering, if a bit of an experimental one. There used to be a variety of toolsets attempting to provide this functionality under Linux (e.g. OpenMosix, OpenSSI) but they all seem to have bitrotted by now.

I almost forgot about those. I'll have to re-read a couple papers.

I have interviewed at one of the biggest retailers in Italy (Esselunga, a retailer in groceries and pretty much all the related things).

While they had they own datacenters (two of them, hosting about ~2k physical x86 server machines and many more virtual machines) all of their retail shop inventory and sales processing ran on IBM SystemZ mainframes. They had more than one of them, and were in the process of buying a new one.

What they told me is basically they have a 24/7 operation with no downtime accepted. They also need to process a huge number of orders (check-out at the grocery stores, basically) in real time. Their inventory systems were not very different from the original ones written in the late 70ies except for some evolutions along the way.

They also told me about that time when they switched from an old mainframe to a new one, during the business day, in the middle of the operations, with no downtime or service interruption.

So basically what I got from that is that the benefits are:

- write once, run forever code

- 100% uptime (not "five nines", literally 100% uptime)

- a reference environment that you can rely on (no runtime change, everything that works today will keep workin in 30 years)

- value added consulting from IBM

- a complete set of tools (runtime, database, primitives, libraries etc)

> - 100% uptime (not "five nines", literally 100% uptime)

There's literally no such thing as 100% uptime. You can add more "nines", meaning that disruptions become exponentially less likely, but that's still very different from "100%".

Oh you've probably not heard of Non-Stop environments for example:

https://en.wikipedia.org/wiki/Tandem_Computers#Tandem_NonSto...

I remember that in late 90's, NonStop was marketed as having some ludicrous (7?) number of nines of uptime, but not 100%.

I think '100% uptime' is pretty much shorthand for "absolutely no reason to turn the machine off or stop services on them running, ever".

As the above link mentions, everything is redundant. CPUs running in (near) lockstep - voting bad cpus out, redundant storage, power etc etc etc.

I've not seen a better write up on IBM's equivalent systems - but I've not had a reason to play with such big iron. NSKs however were much more affordable, and were used in the late 90s/early 2000s in various telcos etc.

I mean, a meteor could hit the DC. But I guess that would count as a decommissioning event, wouldn’t it? So you’re still right, with a proviso—there’s never any reason to turn the machine off with the expectation of ever turning it back on.

Well ... unless you want that exact hardware to move DCs.

Your decommissioning event had me in stitches. I might use that phrase in the future :D

A company I worked for (in São Paulo) had a couple failover setups for its Unisys A-series machine, one with a bank across the street and another with a near twin machine running in its factory in Manaus.

If, for some reason, an event decommissioned all three machines, having our computers back online would be a lesser problem - we would probably be better off learning to hunt and make fire.

Still are in telco, though many are actively being phased out.

Of course there is, it's an insurance buisiness.

A couple of links here:

https://www.ibm.com/support/knowledgecenter/zosbasics/com.ib...

http://www.redbooks.ibm.com/redpapers/pdfs/redp5346.pdf

Plus one that I seem to be posting on a monthly basis these days:

https://www.redbooks.ibm.com/redbooks/pdfs/sg246366.pdf

Here's one I just found, and will be reading tonight I think:

http://www.redbooks.ibm.com/redbooks/pdfs/sg248550.pdf

Enjoy!

Thank you!

No problem :)

If you're really keen, the base z/OS docs can be found here when you've got a spare couple of decades:

https://www-01.ibm.com/servers/resourcelink/svc00100.nsf/pag...

As someone who approached this problem before, the document called "Principles of Operation" IS NOT the correct entry point.

Ha, yeah should probably have mentioned that :)

"As late as 2017, IBM reported 92 of the top 100 banks still used mainframes for their core businesses." https://www.tpr.org/post/how-cobol-still-powers-global-econo...

But why ? Other than inertia and legacy reasons that is. Is there some non partisan resource on this ? A paper or a neutral blog ? Whenever you try to research something like this, you end up on either IBM pages or pages by mainframe enthusiasts. I want to know exactly what you are buying compared to a regular high availability database like setup in the cloud.

I work fairly regularly with our mainframe team, and the general reasoning boils down to massive parallelism and consistency when processing critical data. The number of rows our mainframe can process and write with perfect accuracy is leagues ahead of what similarly priced commodity hardware could achieve. There’s a reason the world still pretty much runs on COBOL/Natural and a hardware design from the 70s, it worked well then and has worked well ever since. Slightly related but I’d describe most of my role being getting data from our micro-services down into the mainframe in a format it understands so we can do ‘something’ with it.

Interesting. So something like google bigquery (or any of the various saas offerings by others) would not be a good match for this problem space ? For massive parallelism, we have GPUs, accelerators, and database products built on these things (admittedly, still somewhat new). By consistency, do you mean tolerance to transient errors and the like ? i.e. Beyond whatever you get with ECC ? Or are you talking about memory consistency models ?

I am just constantly surprised that on the one hand FAANG+ which run so much of the internet economy do not use mainframes, and yet banks do.

The promise of the cloud is that it provides you with "trivially", "infinitely", "cost-efficiently" scalable infrastructure. Banks don't need any of that. They can buy a $100k (?) mainframe, a hundred or so devs, and leave that part of the business on autopilot for 20+ years. Why even bother futzing around in the cloud?

> They can buy a $100k (?) mainframe,

You'll find those on e-Bay. A new one will be significantly more expensive.

Compared to the cost of software engineers a few million every few decades for a mainframe aren't that significant either way.

Also, reaching the same kind of capabilities would require a substantial investment in R&D. IBM has been developing zOS for more than 50 years now.

In addition to what bendbro suggests, which is actually quite close to reality if oversimplified, banks also have sovereignty concerns with data that may have additional classification and regulatory strings attached. In Australia where I’m based, ‘material’ data such as financial and audit information must be kept in country and its veracity attested to, which is why so many large banks here at least still have big iron in managed data centres for most of their core transactional systems.

Edited to specify which sibling I was referring to.

Banks used to store peoples gold, now they store data.

I don't think any serious bank will consider trusting outsider with their core business.

And mainframes are essentially early on premises clouds.

And today's, cloud offerings still don't do what mainframes did as easily.

(and there are several things that cloud does offer that mainframes can't do)

Outsourcing of financial services processing has been a big business for decades. However it is usually provided to small/medium banks or credit unions by specialized firms or by big banks.

A big bank will usually do its own ledger processing. A good configuration is to process in two pairs of data centers more than 200 miles apart in different electric utility interconnection regions, e.g. Arlington/Fort Worth and Dayton/Springfield. Archival storage of transactions may involve additional centers. https://en.wikipedia.org/wiki/IBM_Parallel_Sysplex#Geographi...

Getting distributed systems correct is very hard. Mainframe architecture allows one to program and deploy as if you had a giant computer with consistent transactions for the most part. Since banks were the early adopters of computation in large scale it’s clear why most still run on mainframe - the transition to cloud architecture isn’t a simple translation, you need to glue a lot of things together to the same levels of parallelism, and probably sacrifice some consistency that you get “for free” on mainframe architecture.

Cloud is cheaper ($ per cycle) at the expense of human brain power to glue it all together.

> (admittedly, still somewhat new)

That's the whole explanation right there. "Move fast and break things" is not the way that sort of business works.

Give me a message queue that guarantees exactly-once-delivery to coordinate a cluster of transaction processing mashines and we can talk about banks using commodity hardware.

Someone posted in a related thread I was nerding out on tonight that mainframes are excellent when you need to do an enormous amount of I/O with 100% uptime and "exactly once" consistency guarantees. According to that poster, you also don't need to build distributed logic or certain kinds of error handling into your software.

> But why ?

Recapping what i've written in another comment:

- 100% uptime

- reference, full featured platform that does its job doesn't change much during the years

- write once, run forever approach

- everything on premise

My high level view is that with a mainframe you get a lot of the advantages of the cloud without the hassle of implementing a distributed system / data integrity and consistency over the network (ie something like having the AWS console where you can instantiate services but everything is local. Upgrading the account is physically putting more/swapping resources in the Mainframe). Also there is the cost of migrating, these systems work perfectly reliably since more than 30 years.

AFAIK they have incredible redundancy. Think about RAID-1 but for all components, so one CPU might fail and your software won't even notice that.

It goes a bit deeper than that. I work on POWER which is a sibling to the z platform. They share some common architecture and microarchitectural elements.

To get a sense, have a look at the E980’s Reliability, Availability, and Serviceability Manual. [1] that systems NOT a mainframe - but it borrows ideas from it. Even in that system all levels of it are designed for failure, from Customer replaceable units like fans or power supplies to certain busses have extra lanes or wires for parity or spares depending on the design. Internal to the processor logic there’s a large number of internal checkers that watch the architected state of the machine and notice when something abnormal happens. For several things the processor core will unwind its state and retry an op, and if it proceeds, flag itself as having had a “recoverable” error, which usually triggers a service action. If the system can’t recover it halts either the core or the entire platform, and a dump happens from the service processor which then offloads it to the management console for diagnosis.

Z boxes do all of the above - and more. There’s pictures floating around the office of a system that went through and earthquake, fell over (someone didn’t buy the optional earthquake bracing for the rack) and as it fell, it domino’d over along with several other racks. It still running when the techs went to inspect the data center, though it did turn on its “service me” attention light. ;)

[1] https://www.ibm.com/downloads/cas/2RJYYJML

My favorite advantage is Channel I/O:

https://en.wikipedia.org/wiki/Channel_I/O

That difference a big part of why our PC's and servers have low utilization while mainframes have massive utilization/throughput. There's designed in embedded space copying it, too, where they have a main processor and a cheaper one for I/O.

Reply on news.ycombinator.com