Hacker News Reader

The bulk of my mainframe exposure was with z/VM, but you're right, those systems are fascinating. I've never understood why IBM doesn't provide hobbyist licenses of z/OS and z/VM for individuals to learn on. Of course, the software would be run on Hercules [1] emulator and not a real mainframe. Even so, it would be a great avenue for people to learn about the mainframe software environment and maybe even help with the mainframe skills shortage.

[1] https://en.wikipedia.org/wiki/Hercules_(emulator)

Things were different in the 1970s. And perhaps that explains why IBM is more secretive now.

The Itel AS/5 was a clone of the IBM System 370 Model 158, a very popular mainframe for its time. How was Itel able to clone it? Simple. Just use the hardware schematics!

http://www.silogic.com/NAS/NAS.html

   ALD's were Automated Logic Diagrams. IBM sold these.
   They were complete schematics for the S-370.
   EXSYSCO basically duplicated the logic of the S-370.

I had personal experience with the AS/5. It came supplied with IBM schematics and microcode listings. These were kept in giant binders in the machine room.

When the machine booted up, a Z80 based processor that Itel designed was used to load IBM microcode into the clone mainframe.

Oh, and the application software that ran on that 1970s machine will still run on IBMs latest mainframes.

When IBM introduced the PC in 1981, hardware schematics and BIOS listings were also available for purchase.

Times have changed!

So I joined IBM in 2005 working on firmware for a project called eCLipz, which eventually produced the POWER6 family of systems. We've just shipped POWER9 which is in the #1 and #2 supercomputers in the world, Summit and Sierra. [1]

The "i/p/z" part of the project name is important, because it represents:

i - the lineage of the AS/400, which was a 'minicomputer' design from the 80's designed for multiple users, which came up with the idea of "wizards" for administration tasks before there was even a name for a "wizard." The "i" means "integrated," since IBM i is meant to require very little administration. [2] [4]

p - the lineage of the RS/6000, which begat AIX UNIX. The "p" indicated "performance." The box I worked on shipped at 5 GHz in 2008 (IIRC).

z - the lineage of the System/360 mainframe which the above article discusses. The "z" indicated "zero downtime."

The control structure and power and cooling infrastructure for the systems used common elements with that project. The hardware and firmware underneath the operating systems for i and p converged into Power systems, which eventually produced a variety of projects in the open source community (OpenCAPI, OpenPOWER itself, OpenBMC), and of course runs bi-endian to support things like little-endian Ubuntu or the classic Big-endian AIX or i/OS on the same processor core.

Interesting trivia - In IBM systems (Mainframe or Power), the cold boot of the processor from an unpowered state is called an "Initial Program Load," or IPL. The IPL lingo dates back to System/360 (see its Wikipedia), and survives to this day in really interesting places in the wild. Take the OpenBMC project, which aims to create a 100% open source software stack for those sideband management processors found in servers (and increasingly in other devices too). This github bug from a few months ago which complains about a bug causing "IPL" problems. [3] ;-)

[1] https://www.theverge.com/circuitbreaker/2018/11/12/18087470/...

[2] https://en.wikipedia.org/wiki/IBM_i

[3] https://github.com/openbmc/openbmc/issues/2831

[4] edit - fixed "70's" to "80's" per https://news.ycombinator.com/item?id=18494866 ;-)

I do love the lingo from these machines. I remember encountering the need to "vary off" an interface.

The weirdest was the being told I needed to "wand on" the system. I guess the origin was from the AS/400 when they were pushing cards with a barcode as ids, so you would scan your card with a optical wand (pen) to login to your station. Years later I still hear "wand on".

I have friends who still say their program "abended".

Just a minor nitpick - the green screens we see today are also often descendants of the 3270 family as well as the 5250's that started life with the System/34 and found their way into the AS/400 and iSeries.

I had to nitpick because the 3270 is where this beautiful (shameless plug) font originated: https://github.com/rbanffy/3270font

Ha! Fair enough. Lessee if I can edit...

ABEND was also used in Novell Netware, I wouldn't be surprised if that's the most recent actual incarnation (rather than a reference, as in a game).

Are... are there people who don't? :O

I haven't ever really worked on Mainframe as such, but I _am_ in the IBM/ERP world, and, well, a program Abends, that's just what it does when it goes South. Not even sure what else to call it - failed? Errorred out? ...Segfaulted its guts on the ground? :->

(edit: I suppose I should also fess up that we call virtual machines "LPARs" and physical machines "frames" :P )

It took me a long time to accept Docker's use of the word "container".

To me, "virtual machine" has a distinctive Smalltalk 80 feel. And, of course, it's a bit underwhelming in comparison.

When I first heard about eCLipz, I thought it was pushing towards a single processor architecture for all three series. From today's article, it seems as though z/Series machines still get quite distinct CPUs, though. Do you know to what extend these processors are relatives of the POWER9?

POWER6 and z10 share functional blocks, but the ISAs are pretty different.

I'm not familiar with the z architecture enough to comment with any sense of certainty, and the people I'd normally pester are on vacation, unfortunately.

Thanks for this shot of nostalgia that is rushing through my brain. Yes, I remember IPL from z/OS, working with RACF (Resource Access Control Facility), LPAR (Logical Partitions), and the complexity of Parallel Sysplex (clustering) all leading up to running COBOL and PL/I applications mostly for banks.

The TN3270 terminals, and the wild hacks of writing Assembler macros to translate those TN3270 inputs and outputs to a webservice :-)

I have never worked with the Power systems personally, but heard a lot of good things about it.

It always felt like IBM enterprise stuff that $$$$ heavy are in a world of its own completely disconnected from people who cant think in that much $$$s.

> It always felt like IBM enterprise stuff that $$$$ heavy are in a world of its own completely disconnected from people who cant think in that much $$$s.

You're not wrong. Licenses for even the most trivial of software cost literal fortunes.

It's just that these systems process billions of software and run millions of lines written decades ago.

The cost is a mere blip.

> i - the lineage of the AS/400, which was a 'minicomputer' design from the 70's designed for multiple users, which came up with the idea of "wizards" for administration tasks before there was even a name for a "wizard." The "i" means "integrated," since IBM i is meant to require very little administration. [2]

That specific system was introduced in the 1980s, but it's even more interesting than that, because it's the last remnant of the Future Systems project which did date to the 1970s:

https://people.cs.clemson.edu/~mark/fs.html

> The IBM Future System (FS) project was a design effort to succeed the IBM S/370 architecture that was adopted company wide in 1971. FS was based on a single-level store with automatic data management. The project is reputed to have cost more than $1B when it was cancelled in February 1975, but the single-level store idea was adopted by System/38, AS/400, and iSeries.

It had interesting effects on IBM:

> Most corrosive of all, the old IBM candor died with F/S. Top management, particularly Opel, reacted defensively as F/S headed toward a debacle. The IBM culture that Watson had built was a harsh one, but it encouraged dissent and open controversy. But because of the heavy investment of face by the top management, F/S took years to kill, although its wrongheadedness was obvious from the very outset. "For the first time, during F/S, outspoken criticism became politically dangerous," recalls a former top executive."

Beware the Superproject!

Oh, good point! Yeah, I munged together the AS/400, and its predecessors. And fascinating - I hadn't heard of that history!

Oh - and to avoid rewriting the statement completely, I edited the "70's" to "80's" in the original comment referring to yours.

Was the firmware still based on PL/S? In case you are allowed to disclose it.

Proper IPL is done through switches on a front panel ;)

I actually had to use those switches (well buttons) on an iSeries recently. I’ve never been that nervous with any Unix command.

One bit error and it's back to square one. The good part is that if you do it often enough you start to memorize some of it!

For what it's worth, "IPL" was in use for other sorts of systems in the '80s (doubtless influenced by 360 usage).

Also has stuff like the Language Environment, bytecode based distribution binaries, later followed up by IBM OS/400 as well (now IBM i).

https://www.ibm.com/support/knowledgecenter/en/SSGMCP_4.2.0/...

Basically JME, DEX, MSIL, watchOS bitcode, are modern platforms catching up with mainframes.

They used to provide a free z Linux guest OS available to anyone. I used it to verify that my open source projects worked on Z. It was RedHat 7.3 I think.

Pretty sure it's more that IFLs are not allowed to run z/OS workloads. I think you can run Linux on CPs just fine, it's just more expensive.

I can run a kubernetes cluster, with 1000 of linux images, have them communicate with each other. Hot swap in and out nodes. Such a thing is not out of reach of common folks if you change your mental model of what a computer is. "The network is the computer"

You can also run Kubernetes on a mainframe now, with ZLinux and the hundreds of precompiled s390X (IBM CPU architecture) docker images: https://hub.docker.com/u/ibmcom/.

This is attractive if you happen to already be a large company and locked into mainframe contracts. Sure, you don't need a mainframe to do this, but it's also nice to not need to throw away your mainframe and buy a bunch of commodity hardware to do this.

Options are good for everybody. :-)

But that comes at the expense of complicated software. Sure, once you've gone through the effort of procuring all that hardware, configuring it to run Kubernetes, reconfiguring your software to run in containers... you're good to go.

Mainframes allow you to easily scale your software vertically (but at the expense of complicated hardware). That might seem silly in today's world, but a lot of the software running on Z was written decades ago and the risk of rewriting it is extremely high.

How... does it work then?

Was the programming model for mainframes made to be scalable from the beginning?

The vast majority of mainframe software, especially older software was batch processing oriented - often millions of discreet financial transactions - that can easily be scaled horizontally across multiple CPUs or processing engines.

Coding for a mainframe in the early days was almost regimental in many companies, lots of dotting i's and crossing t's, code sign off, lots of QA and checks.

One example would be a bit of COBOL I did and associated test program, all worked, ticked all the boxes accept there was a single spelling mistake in one comment line, actually I'd missed the full stop of the last sentence. So I had to make that change and retest it all from scratch again. How many would just leave it, or not fully test every instance and associated program that had in full and document all tests and results compared to expected results today? Yes we still have industries that would do that, airline industry or finance systems, but that was the prevail of all code back then.

But then source code control does not consist of a group of auditors in a separate group physically comparing outputs with previous with a lightbox. So been much progress, though equally lots of automation in which a single flaw can cascade.

Now as for scalability - mainframes lived on what we know as moore's law today more than we do today and iirc the likes of IBM effectively promised it's customers a solid path for tomorrow, today and remember - what we now know as LTE today, was and is the mainframe staple and LTE for them, means a lifetime. But you pay for that level of service, but always have. This enables lots of legacy, well tested and proven code to carry on still being used today.

Equally a relevant anecdote, was working in the 80's for a large company who had iirc a DPS88T (Honeywell Bull), which had a maximum of 4 CPU clusters available. This company was hitting max usage and I identified a program that I could optimise and recover near on a whole CPU's worth of processing back for use. I also mentioned that dispite only officially having 4 CPU clusters available, it was possible to attach a 5th. May mistake as next thing they are onto the supplier demanding a 5th CPU. As for them money solved a problem and whilst a cheaper option was available that would entail change and when it comes to changing code that just works as needed, they are always reluctant to change it when they can carry on just adding more power. And the mainframe suppliers make their money on supplying more power, year after year, all in a heavily fault tolerant hardware design that goes beyond just having ECC memory and thinking life is good. But mainframe hardware is a well worth looking into as redundancy, legacy handling, robustness is at a whole level many don't appreciate.

That is an Apples-to-Oranges comparison. The Z-series already changes your mental model of what a computer is if you are not yet familiar with it. A Kubernetes cluster is not a computer, it is a cluster of computers.

The end result (computations being done) may be the same but the way you get there is radically different between those two setups and you're going to have to do a lot more work on your application on a cluster vs a very large computer with some awesome hardware capabilities thrown in. Not cheap, but for some applications well worth it.

> A Kubernetes cluster is not a computer, it is a cluster of computers.

That statement is true, but the thing it implies isn't. A cluster of computers on a common backplane can certainly present itself as a single (NUMA) computer, with one process scheduler, one filesystem, etc. Most HPC systems are built this way.

As well, when your workloads are, themselves, VMs, there isn't much of a difference between having a Z-series machine as a hypervisor and, say, a vSphere cluster as a hypervisor. Either way, you get a pool of CPU, a pool of RAM, a pool of storage, etc. And either way, you get VM workloads with transparent migration on node hotplug/node failure, etc.

> A Kubernetes cluster is not a computer, it is a cluster of computers.

Well, when you tot up all of the channel processors and LPARs and so on...

A mainframe isn't a computer, it is a cluster of computers.

For starters, the software running on a mainframe can include persistent storage.

Well, it is murky if you start generalising that way. The equivalent of Kubernetes in the Mainframe world would be Parallel Sysplex - where there is a cluster of Mainframe servers that co-ordinate with each other and take over upon failure or over load of other participants of the cluster.

A single system Z is vertically integrated and scaled. Now, I was talking about a single operating system that can handle a hotplug out/in of several of its CPUs without any kind of service degradation.

Sure, I can throw in a few x86 servers, and achieve the similar convenience using say load balancer, floating IP, shared storage, and whatever else is needed for a full switch over. You see where I'm getting at. Complexity that is on YOUR plate.

I'm not denying that Kubernetes gives a taste of such resilience for the common folks, just saying that the IBM system Z's engineered solution is on a different level that is just not what Kubernetes is aimed at.

As a side rant, Kubernetes is a magic layer for most of the users, that it is glorious until they hit the first serious problem that makes them scratch their head and look at their complex black box differently. With Mainframes, the banks just call IBM :-)

At some point in history, I had the misfortune of working with Stratus hardware. Not quite where you describe the Z system, but it did operate with hot swappable components, and multiple OS's running in a "fault tolerant" manner.

We hit show stopping firmware bugs that managed to take down every OS. The end result was that the three standalone Dells on top of it had better uptime.

I always wondered how the mainframe world ever managed this. When you market is so small (relative to x86) every case is an edge case. Surely these extremely large and expensive devices still had bugs that transcended that redundancy.

Here's a 100,000 foot view of mainframes (based on when I was working with them and on my limited view).

The mainframes that I worked with (not sure if this is true with all) had something called Capacity Upgrade on Demand. This allowed the data center to temporarily turn on extra processors/engines to handle unusually high processing demands. After the crunch is over, the extra processors can be turned off. I'm not sure how the money aspect of this works.

What the x86 world thinks of as a bare-metal box/chassis is what looks like a full sized rack, except it's enclosed in a cabinet. This bare-metal box is carved up into LPARs (logical partitions) using software called PR/SM [1]. I've read a number of times that PR/SM is a stripped-down version of z/VM. I don't know if that's true or not.

IBM has multiple operating systems for mainframes. The most well known is z/OS (formerly known as MVS). There are others like z/TPF and z/VSE (maybe even more?). There is also z/VM.

JCL (Job Control Language) is used by z/OS (possibly z/VSE too, idk) for running "jobs". This is essentially like shell scripts. The job is submitted and then queued for execution. After execution, the results are made available in an output queue. From Linux on x86 point of view, it's like all program output is being redirected to a file.

There are a number of z/OS add-ons that are typically licensed to run on z/OS. One is TSO (Time Sharing Option). I don't know if TSO is separately licensed or not. Basically, it makes a batch environment almost appear to be interactive. Another component commonly licensed is ISPF (I don't know what it stands for). ISPF is basically a menu-based system that is configurable and I think it comes with some helpful built-in utilities.

z/VM's history goes back to the late 60's and early 70's. Supposedly, one of the primary reasons it was developed was to give z/OS system programmers a sandbox for developing and testing changes. Apparently, some mainframe shops also ran MVS and other operating systems as guests under VM in production (this was probably in 80's). This wasn't the only use of VM though. There was CMS that is a single-user OS that was typically used for end-user business users. Since VM with CMS guests were so common, one would often just see "VM/CMS" written. From within CMS, one could issue CMS commands as well as CP commands. CP is "Control Program" and refers to base z/VM (hypervisor).

One can also run z/VM as a guest under z/VM. This is referred to as 2nd-level VM and is used for testing/evaluating new versions of z/VM before installing them into production.

When Linux hit the scene, versions of Linux were built for mainframes and Linux was added to the list of guest OSs that z/VM supported. To be clear, this is real Linux -- not emulated. It's normal Linux, but compiled to mainframe opcodes.

Once you get access to a Linux guest on z/VM, it's really hard to tell that it's different from x86. It's not weird or different. Any linux user would feel just as much as home with Linux under z/VM as with x86.

Parent posted discussion of CPs and IFLs. Back when I was on the scene, there were also zIIPs and zAPPs. I think zIIP was for encryption or compression (maybe both). I believe zAPP was primarily for running Java workloads. It's been a long time, so my recollection may not be 100% accurate.

Probably the biggest shock to a newcomer of z/VM and z/OS is that terminal sessions are block-oriented (3270) and not character-oriented. With 3270 session, you type whatever you need to on a screen and then when a special attention key is pressed, your screen information is passed to the mainframe to process whatever you typed. No vi editing with 3270. Text editing on 3270 is with line-oriented editors like XEdit.

The scripting language for z/OS and z/VM is REXX. It's as high level as Python and is really a nice language once you get used to its quirks.

Here ends my mainframe brain dump. Can't guarantee that it's all correct or even still relevant, but hopefully gives a little bit of the feel for the environment.

[1] https://en.wikipedia.org/wiki/PR/SM