IBM z14 Microprocessor and System Control Design
fuse.wikichip.org
fuse.wikichip.org
The cool moments I distinctly remember were runing the z/OS installation binaries from tape drives, and swapping in and out - DASDs(Hard disks), and perhaps the coolest was being able to hot plug in/out CPU blades from the cabinet while the system is live.
It's pity that thing is so expensive, and out of reach of the common folks.
For those not in the know, and curious. Based on my knowledge from 2011, CPs are the general purpose CPUs that run z/OS - the main OS companies typically run on System Z. There were special CPUs(may be microcoded so), called IFL that were allowed to run z/VM and Linux. I don't know how things have evolved in the last 7 years :-)
The "i/p/z" part of the project name is important, because it represents:
i - the lineage of the AS/400, which was a 'minicomputer' design from the 80's designed for multiple users, which came up with the idea of "wizards" for administration tasks before there was even a name for a "wizard." The "i" means "integrated," since IBM i is meant to require very little administration. [2] [4]
p - the lineage of the RS/6000, which begat AIX UNIX. The "p" indicated "performance." The box I worked on shipped at 5 GHz in 2008 (IIRC).
z - the lineage of the System/360 mainframe which the above article discusses. The "z" indicated "zero downtime."
The control structure and power and cooling infrastructure for the systems used common elements with that project. The hardware and firmware underneath the operating systems for i and p converged into Power systems, which eventually produced a variety of projects in the open source community (OpenCAPI, OpenPOWER itself, OpenBMC), and of course runs bi-endian to support things like little-endian Ubuntu or the classic Big-endian AIX or i/OS on the same processor core.
Interesting trivia - In IBM systems (Mainframe or Power), the cold boot of the processor from an unpowered state is called an "Initial Program Load," or IPL. The IPL lingo dates back to System/360 (see its Wikipedia), and survives to this day in really interesting places in the wild. Take the OpenBMC project, which aims to create a 100% open source software stack for those sideband management processors found in servers (and increasingly in other devices too). This github bug from a few months ago which complains about a bug causing "IPL" problems. [3] ;-)
[1] https://www.theverge.com/circuitbreaker/2018/11/12/18087470/...
[2] https://en.wikipedia.org/wiki/IBM_i
[3] https://github.com/openbmc/openbmc/issues/2831
[4] edit - fixed "70's" to "80's" per https://news.ycombinator.com/item?id=18494866 ;-)
The weirdest was the being told I needed to "wand on" the system. I guess the origin was from the AS/400 when they were pushing cards with a barcode as ids, so you would scan your card with a optical wand (pen) to login to your station. Years later I still hear "wand on".
Just a minor nitpick - the green screens we see today are also often descendants of the 3270 family as well as the 5250's that started life with the System/34 and found their way into the AS/400 and iSeries.
I had to nitpick because the 3270 is where this beautiful (shameless plug) font originated: https://github.com/rbanffy/3270font
I haven't ever really worked on Mainframe as such, but I _am_ in the IBM/ERP world, and, well, a program Abends, that's just what it does when it goes South. Not even sure what else to call it - failed? Errorred out? ...Segfaulted its guts on the ground? :->
(edit: I suppose I should also fess up that we call virtual machines "LPARs" and physical machines "frames" :P )
To me, "virtual machine" has a distinctive Smalltalk 80 feel. And, of course, it's a bit underwhelming in comparison.
I'm not familiar with the z architecture enough to comment with any sense of certainty, and the people I'd normally pester are on vacation, unfortunately.
The TN3270 terminals, and the wild hacks of writing Assembler macros to translate those TN3270 inputs and outputs to a webservice :-)
I have never worked with the Power systems personally, but heard a lot of good things about it.
It always felt like IBM enterprise stuff that $$$$ heavy are in a world of its own completely disconnected from people who cant think in that much $$$s.
You're not wrong. Licenses for even the most trivial of software cost literal fortunes.
It's just that these systems process billions of software and run millions of lines written decades ago.
The cost is a mere blip.
That specific system was introduced in the 1980s, but it's even more interesting than that, because it's the last remnant of the Future Systems project which did date to the 1970s:
https://people.cs.clemson.edu/~mark/fs.html
> The IBM Future System (FS) project was a design effort to succeed the IBM S/370 architecture that was adopted company wide in 1971. FS was based on a single-level store with automatic data management. The project is reputed to have cost more than $1B when it was cancelled in February 1975, but the single-level store idea was adopted by System/38, AS/400, and iSeries.
It had interesting effects on IBM:
> Most corrosive of all, the old IBM candor died with F/S. Top management, particularly Opel, reacted defensively as F/S headed toward a debacle. The IBM culture that Watson had built was a harsh one, but it encouraged dissent and open controversy. But because of the heavy investment of face by the top management, F/S took years to kill, although its wrongheadedness was obvious from the very outset. "For the first time, during F/S, outspoken criticism became politically dangerous," recalls a former top executive."
Beware the Superproject!
This is attractive if you happen to already be a large company and locked into mainframe contracts. Sure, you don't need a mainframe to do this, but it's also nice to not need to throw away your mainframe and buy a bunch of commodity hardware to do this.
Options are good for everybody. :-)
Mainframes allow you to easily scale your software vertically (but at the expense of complicated hardware). That might seem silly in today's world, but a lot of the software running on Z was written decades ago and the risk of rewriting it is extremely high.
Was the programming model for mainframes made to be scalable from the beginning?
One example would be a bit of COBOL I did and associated test program, all worked, ticked all the boxes accept there was a single spelling mistake in one comment line, actually I'd missed the full stop of the last sentence. So I had to make that change and retest it all from scratch again. How many would just leave it, or not fully test every instance and associated program that had in full and document all tests and results compared to expected results today? Yes we still have industries that would do that, airline industry or finance systems, but that was the prevail of all code back then.
But then source code control does not consist of a group of auditors in a separate group physically comparing outputs with previous with a lightbox. So been much progress, though equally lots of automation in which a single flaw can cascade.
Now as for scalability - mainframes lived on what we know as moore's law today more than we do today and iirc the likes of IBM effectively promised it's customers a solid path for tomorrow, today and remember - what we now know as LTE today, was and is the mainframe staple and LTE for them, means a lifetime. But you pay for that level of service, but always have. This enables lots of legacy, well tested and proven code to carry on still being used today.
Equally a relevant anecdote, was working in the 80's for a large company who had iirc a DPS88T (Honeywell Bull), which had a maximum of 4 CPU clusters available. This company was hitting max usage and I identified a program that I could optimise and recover near on a whole CPU's worth of processing back for use. I also mentioned that dispite only officially having 4 CPU clusters available, it was possible to attach a 5th. May mistake as next thing they are onto the supplier demanding a 5th CPU. As for them money solved a problem and whilst a cheaper option was available that would entail change and when it comes to changing code that just works as needed, they are always reluctant to change it when they can carry on just adding more power. And the mainframe suppliers make their money on supplying more power, year after year, all in a heavily fault tolerant hardware design that goes beyond just having ECC memory and thinking life is good. But mainframe hardware is a well worth looking into as redundancy, legacy handling, robustness is at a whole level many don't appreciate.
The end result (computations being done) may be the same but the way you get there is radically different between those two setups and you're going to have to do a lot more work on your application on a cluster vs a very large computer with some awesome hardware capabilities thrown in. Not cheap, but for some applications well worth it.
That statement is true, but the thing it implies isn't. A cluster of computers on a common backplane can certainly present itself as a single (NUMA) computer, with one process scheduler, one filesystem, etc. Most HPC systems are built this way.
As well, when your workloads are, themselves, VMs, there isn't much of a difference between having a Z-series machine as a hypervisor and, say, a vSphere cluster as a hypervisor. Either way, you get a pool of CPU, a pool of RAM, a pool of storage, etc. And either way, you get VM workloads with transparent migration on node hotplug/node failure, etc.
Well, when you tot up all of the channel processors and LPARs and so on...
A mainframe isn't a computer, it is a cluster of computers.
A single system Z is vertically integrated and scaled. Now, I was talking about a single operating system that can handle a hotplug out/in of several of its CPUs without any kind of service degradation.
Sure, I can throw in a few x86 servers, and achieve the similar convenience using say load balancer, floating IP, shared storage, and whatever else is needed for a full switch over. You see where I'm getting at. Complexity that is on YOUR plate.
I'm not denying that Kubernetes gives a taste of such resilience for the common folks, just saying that the IBM system Z's engineered solution is on a different level that is just not what Kubernetes is aimed at.
As a side rant, Kubernetes is a magic layer for most of the users, that it is glorious until they hit the first serious problem that makes them scratch their head and look at their complex black box differently. With Mainframes, the banks just call IBM :-)
https://www.ibm.com/support/knowledgecenter/en/SSGMCP_4.2.0/...
Basically JME, DEX, MSIL, watchOS bitcode, are modern platforms catching up with mainframes.
We hit show stopping firmware bugs that managed to take down every OS. The end result was that the three standalone Dells on top of it had better uptime.
I always wondered how the mainframe world ever managed this. When you market is so small (relative to x86) every case is an edge case. Surely these extremely large and expensive devices still had bugs that transcended that redundancy.
The Itel AS/5 was a clone of the IBM System 370 Model 158, a very popular mainframe for its time. How was Itel able to clone it? Simple. Just use the hardware schematics!
http://www.silogic.com/NAS/NAS.html
ALD's were Automated Logic Diagrams. IBM sold these.
They were complete schematics for the S-370.
EXSYSCO basically duplicated the logic of the S-370.
When the machine booted up, a Z80 based processor that Itel designed was used to load IBM microcode into the clone mainframe.
Oh, and the application software that ran on that 1970s machine will still run on IBMs latest mainframes.
When IBM introduced the PC in 1981, hardware schematics and BIOS listings were also available for purchase.
Times have changed!
The mainframes that I worked with (not sure if this is true with all) had something called Capacity Upgrade on Demand. This allowed the data center to temporarily turn on extra processors/engines to handle unusually high processing demands. After the crunch is over, the extra processors can be turned off. I'm not sure how the money aspect of this works.
What the x86 world thinks of as a bare-metal box/chassis is what looks like a full sized rack, except it's enclosed in a cabinet. This bare-metal box is carved up into LPARs (logical partitions) using software called PR/SM [1]. I've read a number of times that PR/SM is a stripped-down version of z/VM. I don't know if that's true or not.
IBM has multiple operating systems for mainframes. The most well known is z/OS (formerly known as MVS). There are others like z/TPF and z/VSE (maybe even more?). There is also z/VM.
JCL (Job Control Language) is used by z/OS (possibly z/VSE too, idk) for running "jobs". This is essentially like shell scripts. The job is submitted and then queued for execution. After execution, the results are made available in an output queue. From Linux on x86 point of view, it's like all program output is being redirected to a file.
There are a number of z/OS add-ons that are typically licensed to run on z/OS. One is TSO (Time Sharing Option). I don't know if TSO is separately licensed or not. Basically, it makes a batch environment almost appear to be interactive. Another component commonly licensed is ISPF (I don't know what it stands for). ISPF is basically a menu-based system that is configurable and I think it comes with some helpful built-in utilities.
z/VM's history goes back to the late 60's and early 70's. Supposedly, one of the primary reasons it was developed was to give z/OS system programmers a sandbox for developing and testing changes. Apparently, some mainframe shops also ran MVS and other operating systems as guests under VM in production (this was probably in 80's). This wasn't the only use of VM though. There was CMS that is a single-user OS that was typically used for end-user business users. Since VM with CMS guests were so common, one would often just see "VM/CMS" written. From within CMS, one could issue CMS commands as well as CP commands. CP is "Control Program" and refers to base z/VM (hypervisor).
One can also run z/VM as a guest under z/VM. This is referred to as 2nd-level VM and is used for testing/evaluating new versions of z/VM before installing them into production.
When Linux hit the scene, versions of Linux were built for mainframes and Linux was added to the list of guest OSs that z/VM supported. To be clear, this is real Linux -- not emulated. It's normal Linux, but compiled to mainframe opcodes.
Once you get access to a Linux guest on z/VM, it's really hard to tell that it's different from x86. It's not weird or different. Any linux user would feel just as much as home with Linux under z/VM as with x86.
Parent posted discussion of CPs and IFLs. Back when I was on the scene, there were also zIIPs and zAPPs. I think zIIP was for encryption or compression (maybe both). I believe zAPP was primarily for running Java workloads. It's been a long time, so my recollection may not be 100% accurate.
Probably the biggest shock to a newcomer of z/VM and z/OS is that terminal sessions are block-oriented (3270) and not character-oriented. With 3270 session, you type whatever you need to on a screen and then when a special attention key is pressed, your screen information is passed to the mainframe to process whatever you typed. No vi editing with 3270. Text editing on 3270 is with line-oriented editors like XEdit.
The scripting language for z/OS and z/VM is REXX. It's as high level as Python and is really a nice language once you get used to its quirks.
Here ends my mainframe brain dump. Can't guarantee that it's all correct or even still relevant, but hopefully gives a little bit of the feel for the environment.
BTW, mainframe group takes reliability very seriously. Basically it must not ever corrupt customer (financial) data plus the fit rate must be as good as it can possibly be. I remember when they were making the zBx add-on (an x86 blade center for z which could run Java web apps), they tested the x86 in a cyclotron beam to get an idea of its fit rate- it was quite good (good enough), but not as good as z's CPU.
It can detect, correct and survive normal bit errors of course. But then it can detect, correct and survive power regulator, clock, DRAM, lane, controller and channel failures. RAIM (Redundant Array of Independent Memory) eats up 20% of the memory capacity.
If you want Z CPU, then you might want the memory subsystem as well. If you have them, you probably want the redundant IO as well. Then it's easier to buy the whole thing.
seriously people underestimate the cost to move off a platform that is working and worse when there are so many touch points no one can seem to get an accurate handle on them all.
IBM does indeed take their reliability seriously, both i and Z are all built around not having any downtime but the Z does do it better than i which still can require it for patches.
i just find it fascinating the work that we use these machines for and how instead of being replaced more work lands on them because of reliability issues of other platforms. That and support requirements of other platforms. The z and i teams are three man admin with a few operators to nod heads... the other platforms seem to have armies
I find mainframes fascinating beasts, I don't understand how a programmer would be able to make use of all those "CPs" but it sure looks like a lot of power to use!
It's very exotic, maybe too exotic. I'd imagine hiring junior people and training them up on these things is a lot of effort and probably very hard to hire for - in competition with the companies offering cloud experience etc
IBM actually holds a very good week long training you can send junior developers to. It’s basically a boot camp. There were about 10 other developers at it when I went, most of whom were fresh college grads from the big mainframe software companies (BMC, CA, Rocket, etc.).
I’ve since moved on to the world of .net and I don’t miss the mainframe stuff at all.
Edit: I’ll add that a lot of what made it difficult for me was that I was working with assembly language and our product required a lot of knowledge about the OS. Many mainframe programmers are using java and eclipse, which isn’t that different from using java and eclipse to target any other system.
Think of those big tower PCs of yore, just a big larger.
Single-thread performance is also pretty high, so it'd only lose in the aggregate CPU benchmarks.
Not true. If you view the network as the computer, then the network is now forever winning.
What is Google's or Facebooks uptime? Their computer is their cluster. Those clusters are never going to be powered off ever. They can upgrade and replace the entire cluster without being down. If you have a cluster of 10,000 nodes, turning off 100 or 1000 is not downtime, so longer as it's up and serving. No mainframe ever will run for the next 100 years. It's a possibility that a cluster built today could be running for the next 100yrs.
It's harder to get mainframe like uptimes for OLTP simultaneously with mainframe like IOPS.
99.9 percent. Cloud services from Microsoft, Google, Amazon typically have only three nines or less for paying customers. They also seem to have same level of uptime in their own services.
But these are not "firm guarantees". For example Amazon just gives service credits after downtime exceeds what is promised. They "undergo unscheduled outages" from time to time. Whole regions are down at once.
Remember Amazon's Prime Day outages? It's not good for business when tings go down.
And I bet all that costs much more than an IBM. Not a computer, the company.
The z14 has a ton of cache and does everything it can for single threaded performance, but for the money you could buy a ton of cores (Intel Xeons or AMD chips) and trounce the mainframe on benchmarks that can use the parallelism.
For the money, you would be able to buy the gear to have a lot of bandwidth between nodes and to the storage.
IBM sells it with moniker "Pervasive Encryption" but if you look under the hood, it is actually impressive and very detailed.
Build in security at the hardware level. Especially the Crypto Express6S card. There is hardware acceleration for everything cryptography (including SHA-3 and SHAKE), tamper sensing, true random number generation. Encrypted paging in z/vm, etc. SSL/TLS can run fast. Security auditing is not an afterthought.
Of course you can add all this in special cards for x86 server racks but the cost is probably the same with lower IO throughput and less integration. Getting the same uptime in production would be significantly more expensive if not impossible.
One thing that has been learned is, software will sometimes do interesting things if presented with so much memory. For example: https://github.com/zfsonlinux/zfs/issues/7275
The SGI Altix line, which used Itanium 2, could go up to 1024 cores with a single system image, which is just insane. We had one of those at Goddard a decade or so ago, but I didn't have access to it.