ABCs of IBM z/OS System Programming Volume 1 | Hacker News Reader

ABCs of IBM z/OS System Programming Volume 1

redbooks.ibm.com

If your software stack has per core licensing, a z/architecture mainframe can make a lot of sense. They are extremely efficient at running virtual machines (with linux guests for example). Where an Intel cluster can handle 10-20 VMs per core, a z/arch mainframe can do 100s per core.

They have dedicated processors for offloading IO operations, which has a big impact on reducing pipeline stalls, and also part of why they are so much better at handling a multitude of VMs.

They have 4 levels of cache, with gigabytes of memory at level 4, data is kept close to the processors.

The hardware costs a lot, but if your software is licensed per core, there is a good chance you could save millions compared to an Intel solution. It depends on your usage scenario though. If everything is open source and you aren't paying per core fees, then there is probably no advantage for you.

Who licenses per core but then ignores virtualization? (For apps where you aren't throughput bound.)

cough Oracle cough

Just be careful with this strategy. When you could start offloading per-socket IBM licensed software to more performant multi-core Intel or POWER CPUs, IBM switched from socket to "Processor Value Units". (Aka "give us more money")

A single core is not equivalent to a hexa core so I think PVU's make more sense in today's world. What I would like from them is to be more transparent with the requirement to use ILMT.

Does anyone know what kinds of companies or applications make use of z/OS mainframes?

Every large financial institution, airline, or large company that does millions of transactions an hour that isn't competing with IBM or young enough to implement a different platform that can scale to that level.

There is a z/OS security talk I saw that partially dispels the myths and prejudices against mainframes as legacy. In some cases z/OS is more modern than what you might find on your desktop and OSs like Linux are implementing features in its kernel that have been in zOS for years.

That would be Soldier of Fortran - https://www.youtube.com/watch?v=Xfl4spvM5DI

Right I love his stuff. I was thinking of his talk at BSides LV 2014 which he gave the same talk but he also had talks this year also at BSides and DefCon.

I also wanted to mention that where I've worked and many other places there is a crisis of brain drain at companies as the mainframe experts who are highly paid retire out and there are programs to recruit young people who are more attracted to other platforms. There is a lot of pay and job security in the field. I know of one situation where a guy past retirement is paid full salary to work 1/3 of full-time from his vacation home because the company could not afford to lose his knowledge and struggled to find someone competent enough to learn under him that was interested in mainframe. In other words, someone young with these skills would be highly paid and in higher demand at many companies.

Is there any way to learn this sort of thing outside of being at a company with a z/architecture mainframe? I know about the Hercules project, but as far as I can tell, you can only run predecessors to Z/OS (due to licensing as I understand it), and zLinux.

Yep. IBM has been running a hackathon since early 2002 all around the world as a recruiting funnel to resolve the talent shortage problems @surge refers to. Runs every year, all across the world and there are always great prizes (mainframe tshirts, ipads, iphones, macbook pros) just for completing the exercises plus side-track options to drop out of uni and instantly obtain six-figure employment at one of IBM's customers if you do well.

http://www-03.ibm.com/systems/z/education/academic/masterthe... and https://www.facebook.com/MasterTheMainframe

A million tx/hour is only few hundred/second. I've done call routing, including full ACID balance updates, at over 10x that volume, on commodity hardware and software. Why is this a huge challenge for anyone?

I get the feeling it's mostly legacy plus scenarios where the rigor IBM applies is valuable and you go along with whatever they suggest.

It depends on the size/type of the transaction of course... A financial transaction might not be exactly comparable with a call setup.

IBM doesn't have magic pixie dust in zseries mainframes. I had one, and trust me, you can beat it in every single metric with commodity hardware.

What makes them "faster" in many cases is that the decades old apps were written by people "with a clue" and they run pretty much on the hardware. I've had a joke going for a few years, that websphere was created to cause efforts to port off the mainframe to fail.

AKA its pretty much impossible to take something running in ztpf, rewrite it in java and plunk it on a EC2 instance and have a hopes chance in hell of making it work without spending more than the next two decades of mainframe maintenance costs in monthly amazon bills.

That doesn't mean you can't rewrite it in C++ on a DL580 with an attached Flash840 and leave the mainframe in the dust if you have competent programmers.

> you can beat it in every single metric with commodity hardware.

The zSeries selling point is not the hardware, but the reliability (which is in fact the hardware). The z in that series stands for zero down time and they are not selling hardware, but close to 100% reliability.

Now in reality, no one can guarantee zero down time, but IBM does in fact guarantee 5 nines (99.999%) and 6 nines (99.9999%) up time and that guarantee costs money.

You certainly can't get those sort of up-time numbers using nothing but commodity hardware.

Sure. But I doubt it's more than an order of magnitude more difficult. Each call needs routing info (lookups over a hundred million rows a few times). It also generates a CDR, which, in some systems, means a full transaction (insert a row, update a balance).

Another way to view this: since mainframe workloads have existed for a long time, if their performance was way beyond what you can still accomplish with commodity hardware, then we should see tech companies utterly flocking to these systems. We should see IBM selling their chips all over (after all, why wouldn't Microsoft want to get in on the action?). No one wants to deal with huge fleets of servers. If they could just give IBM some money and be done, they'd do that, no?

But instead it's mostly used for older, risk adverse, applications. So while I've no doubt they end up delivering solid systems, I'm skeptical that the scale is more massive than people are used to.

Many retail (brick and mortar chains) use them extensively. Way back in the day, that was what you did when you wanted to process "big data" in the 80's and 90's. Those systems became so ingrained into the business processes of the retailers that they are hard to remove.

I've seen some retailers in the North East try and "get off the mainframe" by moving to a modern ERP from Oracle or SAP only to have the projects fail after spending $250m-$500m and to this day they are still running 20-40 year old backend software for things like financials, pricing, payroll and merchandising. I've seen production COBOL code running that was authored in the late 70's.

I was on the outskirts of a conversation where someone wanted to get away from the "inflexibility of the mainframe", so they were moving to SAP. I missed the rest of the conversation, because I was choking on my beer.

Lmao. I would've been to. Like with person you replied to, retail companies in my area still use mainframes, AS/400, and even IBM's DOS variant. Few people even understand fully how it works and they're not young people wanting a ton of change to happen. Improvements are incremental and stay within the terminal.

However, they do use newer technologies on GUI's etc that integrate with their mainframe solutions. People often have to keep working with two, inconsistent systems at once, though.

Almost every Fortune 500 and most branches of the federal government.

There are a few exceptions obviously. Microsoft, Apple, HP, etc. are unlikely to use IBM technologies internally. Facebook, Amazon, etc. are young enough to not have mainframes.

You can go search for z/OS job openings and see everyone from Walmart to Lockheed Martin to Citibank.

Microsoft did get called out for using a mini-mainframe, the AS/400, back when they were pushing Windows Server. Reportedly they ran their entire business on one AS/400. Embarrassed, they got off the AS/400. Took about 20-23 Windows servers to match its performance and reliability. Told me what I needed to know. ;)

But, yeah, they don't use it any more. Most mainframe users are legacy users. However, there's occasional conversions especially for consolidation. IBM's competitor, Group Bull, posted some case studies on that such as consolidating hundreds of desktops and servers onto 2 easily-managed mainframes with cost savings.

I do remember this event, when Microsoft was forced to admit they did not use any of their their own technology other than the desktop and it resulted in a big hit to the credibility of any the Microsoft server technologies.

But the result actually meant Microsoft had to then eat their own dog food and in a strange way it forced Microsoft to greatly improve their server platforms to where they are today.

So in a funny way it probably backfired on those early Microsoft detractors.

Never thought of it like that. Damn. Why did we have to run our mouths lol...

The State of New York uses them. Once a law was passed that the state needs to turn paperwork around in less than 48 hours they found the printing system was a bottleneck so they hired a greybeard friend of mine to build a new printer interface and device drivers for it.

Cornell University also used to run HR and accounting on a 370 mainframe and people were happy, but then they switched to peoplesoft and now (i) they are not happy, and (ii) looking for a new accounting system.

All US states, 90%+ of the big banks, retailers, airlines, brokerages, etc use mainframes.

There are still places using non-IBM mainframes as well. Several state welfare systems use Unisys mainframes.

Unisys, Fujitsu, and Group Bull that I recall. Too bad Unisys bought Convergent's CTOS. Was the only thing close to Plan 9 that had big take-up in business. I'm not counting Inferno because we're talking mainframes and big business.

Financial services are a good example. Lots of highly regulated legacy systems.

For many big name banks and credit cards, it's pretty safe to assume that a Z/OS mainframe is doing some work with the data you've generated with every swipe.

Nationwide did a big linux on z consolidation a couple of years back.

I worked for a company that does z/OS software and got to see their client list--it was pretty much a who's who of industry leaders in every industry, plus a bunch of colleges.

Royal Bank of Canada for sure. Other banks from Big 5, probably too.

In my experience: Insurance, banking, credit card processing, airline reservations, payroll processing.

"... a 13-volume collection that provides an introduction ..."

What's the z/OS jobs scene like now?

Good, in that many of the folks who know z/OS programming are retiring or dying.

Best time to get in it. Learn mainframe, CICS, RPG, and COBOL. Among these are many jobs that work steady hours with pay that stays the same or increases over time due to lack of labor. Boring, predictable work. Pretty much the opposite of IT in general. And remember that you can always do consulting or FOSS in modern tech after hours for fun. ;)

What do you base this on? There are also tens of thousands of layed cobol programmers who are unemployed or working in other fields (in the US), the legacy of offshoring and cost reduction of the past 15 years.

I am one. I started mainframe cobol in 2003, grew the skills you listed for 5 years. It's a miserable existence for anyone really interested in developement. The company I worked for had just started their off shoring program when I switched to JEE (I took a pay cut, but am way ahead now). That same company subsequently laid off most of their onshore developers.

This is mainframe applications programming today, there maybe jobs, but not in the US.

There's job adds all over the country plus people retiring. Personally, I think it varies area by area. Places with a lot of big companies with legacy systems need employees. Others... not so much.

The offshoring issue is real. It's why most of America's brightest in CompSci and Math are going to financial sector instead of IT. Any time I meet someone trying to get into IT I recommend against it. Only exception, which is risky, is embedded where the need to physically be there reduces offshoring risk. ASIC's, too, so long as you get out of RTL work as quickly as possible: it's all going to Asia.

I'm not closely involved with it, but my understanding is that there is a big push to train software engineers in India on z/OS to make up for aging out of talent (i.e. retiring).

The people in India are cheap compared to the aging IBMers in US. The push is economical and forced. They could easily make the same push here, but it won't happen in the current environment.

That's not true: there is a push here. There's been previous articles, college partnerships, and so on. I'm not going to pretend they'll put more effort here than on cheap labor. However, there's been colleges teaching COBOL, mainframe, etc, even community colleges, with almost nobody trying to get into those jobs outside people already working one. This is one of few times where there might be a real labor shortage as one generation retires or dies off without another to take the reins.

I talked to a professor at a community college teaching mainframe skills a couple years ago.

I asked him point blank at one point, what the average starting salary for a graduate from his school hired to do mainframe work was. The number he threw out was less than 1/2 of the starting salary of the average comp-sci graduate from the state school down the road. Granted the people from the CC had a two year degree, and were being hired for pretty low level positions. But unless the pay ramp was really steep anyone doing it was crazy not to just enroll in the comp-sci department at the state school and put in the additional two years.

I considered if becoming a mainframe expert was a good career move, and from what I can tell midlevel mainframe guys don't make as much as midlevel PHP experts. The top end is ok, but i've heard of people doing vmware designs/installs for more money. Maybe some mainframe guys are paid by the gold bar, but I sure didn't meet any.

That is probably why there is a "shortage". People my age look at it and decide its not a good move because while companies will pay 8 figures for a machine they won't pay a fraction of it to the guys running said machine.

I agree with you. I've spent few years being a mainframe monkey (support). I got out when I had the opportunity. I have seen how people in other countries are taking on this role (of being mainframe monkeys) like there's nothing else in the world. This is mainly because they are cheap labor. While at the same time, the people in the States doing the same, are on the edge of their seats with minimum wage (ask MF operators).

The high-end MF programmers are far and few. Most of them, are there in a unique position within any company (there are hardly two of the same kind within the same company). They are mostly pricks and aholes in themselves (because of their position, their ego makes them as such). But that's a side topic.

Irrespective of what we see with the collaboration in educational field, the MF jobs are just not economically appealing enough for US graduates. Indians, Brazilians and Chinese, on the other hand, have that opportunity because American companies are getting them cheap comparatively.

I think the distinction here between low and high-end MF operators may not be just that they're MF operators at certain experience level. At a number of companies, I've seen that the people on top have a thorough understanding of the applications the mainframe supports and are critical for making changes that don't break it. Not having them any more could make stuff stop working. So, it justifies their higher pay plus job security.

You think that might be the case for the high-end MF operators in general?

The good operators stay long, but they are still on the edge. And there's nobody whose going to tell you how valuable those good operators are for any company's day to day operations. But again, they are not equally valued. What ultimately matters: can the Indians do the job? The answer is yes. And that's where there is cost saving.

I guess the job security is down to what my old pal told me. He was an embedded engineer partly because it was interesting and partly for job security. I asked what he meant about job security. He said his work needed people to physical be there, get the job done, and support it over time with quick, physical response if needed. Harder to offshore than code and only so many H1-B's they can do. So, he predicted we'd have a lot more to worry about.

He seems right so far. Most of the embedded industry overseas is production or low-level stuff easy to hand off. There's usually someone like him over here involved in one or more parts of the process. Maybe I should've picked that market.

There are American companies out there who do not have to worry about H1-Bs because their whole business is being run from other countries. They just shift the workload altogether, with mid managers etc, leaving only the highest of the hierarchy in the US. As a low level tech, I have been through several offshoring where I have seen the whole local departments disappear, only to reemerge in another country with new sets of people. I managed to survive for some time because they could not offshore due to some government regulations. The truth is, American companies will offshore as much as they can. They have and they will. It is a simple case of increased profits due to cost-saving.

Makes sense.

I think there's a distinction to made in this discussion between mainframe application programmers (ie cobol, anyone taking intro courses at an IBM sponsored school) and system programmers ie the guys setting up and writing site integration stuff ie the guys these books are ttargeted toward. Similar to php vs people writing drivers in your example.

Unfortunately this distinction is rarely delinated in these threads.

I didn't make that distinction. Good that you brought it up. Any more information on that distinction in form of articles or salary surveys?

Appreciate your perspective on that. Curious, what was the cost of living in that area? Mine is one of the lower in the country: $50-70k/yr/household is decent living. People here often see what people get paid in Bay area, New York, Chicago, etc and think we're hugely ripped off. Then, they look at housing, etc to see a bit different picture.

So, I wonder if it's that industry segment in general getting ripped off or just your area.

The area in question was a large midwestern city, so its not exactly tops, but the pay rates for computer people in that area are almost always >$100k for anyone with experience.

I don't have a breakdown of payrates for the CC in question, but I'm guessing many of the students going into plumbing/electrician/AC repair/etc were making as much or more than the students going into the mainframe positions.

Ouch...

Same thing in Europe.

People are picky what technology to work on, whereas Indian consulting companies will just work with anything they can get their hands on, regardless of the technology or pay.

Airline GDS such a Sabre are on such systems.

AFAIK, SABRE is/was an airlines reservation system, not a GDS. SABRE evolved into TPF which is not related to z/OS except inasmuch as they are both operating systems that run on IBM z-series mainframes.

See http://www-03.ibm.com/systems/z/os/ for a list of IBM z-series operating systems.

I worked with SABRE -- it's a GDS, and a reason I really want to avoid the entire industry.

SABRE is a GDS - it's used by a whole bunch of US and non-US airlines.

Looks like an uphill, learning experience.

And frustrating for anyone with extensive experience with modern software stacks. The zseries is a strange combination of semi-modern hardware with ancient software/hardware practices. I spent a year of my life with a recent mainframe as part of my previous job. During that time I discovered a long list of things that are frankly unbelievable to anyone familiar with modern computing. I should write a book about all the crazy crap, but I don't really have too. Just reading the install/setup/operating guides with a mainframe in hand should be sufficient for anyone with a clue. Bonus points if you install any industry standard benchmarks to measure things like memory/disk io/networking latency/bandwidth.

But, in the end I get it. People talk about all kinds of crap that isn't true about the mainframe, but that doesn't matter. What companies are getting when they buy a zeries from IBM is a software stack with a 99.99% future compatibility statement. That means the millions you invest in developing software today will still work two decades from now. IBM isn't going to pull the rug out from under you when the latest OS/language/etc shows up. That is why its possible to run JCL/COBOL/etc written decades ago without change on modern machines. For companies that use computers to assist their main line of business (rather than being the main line itself) this is an extremely valuable feature.

"What companies are getting when they buy a zeries from IBM is a software stack with a 99.99% future compatibility statement. "

That is the greatest benefit. I think there's a potential market for a company using something modern (and maintainable) that offers same benefit. Especially if it can work side-by-side with mainframes and/or popular stacks.

Reply on news.ycombinator.com