IBM unveils z-series with 5.5 GHz CPU and hardware transactional memory.
www-03.ibm.com
www-03.ibm.com
Basically it's a different universe. Everything that might be familiar to someone with a unix or windows background is ... different. More eldritch. More powerful and yet more obtuse.
Oh and JCL. JCL is the devil's work.
One thing they did that I thought was cool was to consolidate a bunch of Linux servers onto their mainframes. They replaced a few racks with z/VM instances. If they needed another server, they could spin it up in literally seconds if the template was already loaded. This was back in 2004, 2005.
You don't buy (actually: rent) a mainframe (actually: CPU cycles and RAM capacity) because it's cost-effective. You do so because the system absolutely must work and you have a mind-numbing number of transactions to process in a hurry.
My understanding is that these machines are amazing at IO, and handle virtualized environments incredibly well (and have so for decades) (e.g. thousands of Linux machines running).
If you need raw CPU performance then a mainframe is not for you. If you need competitive pricing then a mainframe is not for you. If you have a mainframe and need a have an expiring agreement then this machine may be for you. If you have legacy mainframe code then this machine may be for you.
I remember watching a financial update from Sam Palmisano (IBM's previous CEO). I swear he started to drool when referring to IBM's mainframe margins. From my experience mainframes are for legacy companies with legacy systems, as they are not price competitive with other markets that experience real market competition. But, it is interesting and incredibly reliable hardware.
What makes them amazing at IO?
Isn't IO more a result of what type of disks a computer has? An SAS 15k disk can only do 300 iops whether it's a mainframe or a commodity server, no?
Lots of cache memory and very fast buses to very large external disk arrays. An old mainframe trick was to use the previous generation hardware as the IO controller for the current generation...
And this is a theme replicated throughout mainframe design. Everything is replicated, all pathways are wide as possible, everything is integrated from the hardware up to the application layer.
Take virtualisation, for instance. IBM mainframes have PR/SM, a hardware-level capability to virtualise everything in the system down to the lowest level of granularity. And they've done this since 1985.
A SATA port can do some number of IOs per second (this depends on how the controller is designed for the most part) and the SATA data pipe is 6 gigabits/second or 600 megabytes per second these days.
A SATA Disk (the spinning rust kind) is different, it can pump out some maximum amount of data per second (usually about 100 Mbytes/sec and has an average seek time of a few milleseconds. So if you are doing random i/os to the disk, so that every I/O needs a seek, and the seek takes 10 mS, you can do at most 100 I/O operations per second (IOPS) if you transfer 512 bytes with each IOP you get a throughput of 100* .5K or 50K bytes per second. If you transfer 100Kbytes per transaction you get 10 megabytes per second. Anyway the bottom line is that an I/O channel is bound both by its bandwidth and the number of IOPS it can do.
A PCIe 'lane' can do 250Mbytes per second and with Intel's 5500 chipset about 2 million IOPs. (Although those IOPs are spread across all PCIe channels) A "big" server chip set will give you 32 or maybe 48 PCIe "lanes" which can be 12 by 4 (or x4) lanes or 2x16 + 8x1, or 2x16, 1x4 and 4x1 (this is a common desktop config for two video cards and misc peripherals)
The bottle neck is exactly like networking, each I/O device can consume a certain amount of channel bandwidth, since they don't run 24/7 you can "oversubscribe" which is to say potentially have more I/O devices than you have bandwidth to service them (this happens a lot in like big disk arrays) and ultimately the transaction rate of the entire system is constrained by the cross sectional bandwidth of memory and I/O channels such that you get I/O limited in terms of things you can push around.
Anyway, a mainframe has thousands of channels. Lots and lots of them. And each of those channels has a lot of raw bandwidth and can sustain a high transaction rate. This makes a mainframe a transaction monster.
In the taxonomy of computer systems, 'supercomputers' have a monster amount of memory bandwidth and high transaction rates (low latency) between their compute nodes (or cores). Mainframes have a monster amount of I/O bandwidth with high transaction rates between their nodes and I/O devices. General purpose computers achieve good economics by limiting memory and I/O bandwidth to stuff that will fit in a single piece of silicon. (or at most a couple of pieces)
Of course, mainframes are also faster simply because they are more expensive (e.g. 100 PCIe slots are obviously faster than 10 slots), but that's not an intrinsic advantage (in theory you could develop a $10M x86 server — call SGI if you're interested).
That said, they do deal with IO pretty well. They have dedicated offload processors for transferring data, which means for the same (IO intensive) workload, the CP utilisation of a zSeries machine would be much less than that of an x86, pSeries (or other) machine. And in terms of the disk being a bottle neck... If you're pushing to a disk array that has 192GB (pretty standard) of write-back cache... then no, the disks are not a large concern.
But... At the end of the day, does this really weigh up to the cost of Big Iron, or for that matter the additional licensing of software on top of it?... Not really.. is my answer.
My father worked on a product called SnapVantage in the late 90s and early 2000s, which basically used the copy-on-write abilities of the virtualized mainframe disk in a StorageTek disk array combined with z/Linux and z/VM as a hypervisor to instantly provision and boot Linux machines.
It looked like a very, very toned down version of the AWS management console, with a near-identical ability to select a distribution, upload SSH keys, find an IP, and have a full, running VM in seconds... but this was in 1999, when watching a Linux box "copy," provision, and come up in just a few seconds was mind-blowing.
I used my nascent Perl skills (as a ten-year-old) to write the installer for the host software, which I believe actually shipped to customers nearly unmodified!
The zSeries still supports a lot of battle-hardened HA features like lock-step execution, hot CPU fail-over, and hot-swap memory and IO which are very rare, esoteric, and hence hard to support in the x86 world (and rare, esoteric, and untested is not what you want in your HA system).
Plus, the number of VMs that can run well on a z/Linux and z/VM environment is amazing - we had hundreds and hundreds of instances running on one zSeries.
There's a very full-featured zSeries emulator called Hercules which will allow you to run z/OS and z/VM (if you can find them, of course). There's also a distribution of MVS 3.8j, the 1980s System 370 / System 390 precursor to z/OS, available for Hercules for free if you want to get a taste of what administering such a system is like. A lot of obscure job-scripting languages like JCL and early scripting languages like REXX were developed for early mainframes and still live on today.
I can't recall any specific stats or stories right now but you can imagine the discussions between a father who develops primarily with Assembly and a son (me) who ended up being a .NET developer working on various startups. We've had a few talks about the 'cloud' mostly resulting in him laughing at what we call 'web scale'.
TPF is an OS, transaction processing & database system that was written in mainframe assembler and deployed in 1979 to run SABRE and a few other core transaction systems (VISA, AMEX, Marriott hotels, etc.) - maybe 100 customers still use it today - and it was a descendent of a previous assembler system developed in the 1960's.
It probably was (and is) the closest system to "web scale" through the 80's and 90s's in that it ran some of the highest volume workloads globally (e.g. 5000-10000+ tps back in the 80s / 90s, 25000+ tps today) at 99.99% availability. In context, Twitter hits 10-15k tweets per second during big sports events.
These days it's extremely expensive and slow to make changes to TPF relative to modern technology (as it traditionally required S/370 or S/390 assembler, though these days requires C). IBM has a newer version called z/TPF which makes it easier to code in C or C++ and communicate with the outside world via web services, etc.
A very different world...
maybe 100 customers still use it today
Indeed it does seem like there are only a few companies that use this stuff. Like you said, you had to be a pretty big company with massive amounts of transactions to merit it. He's actually contracting with the IRS right working on some newer systems to handle your tax returns. (He tells me every year it's a miracle you get your tax returns every year on the old system)
Whenever he's looking for a new gig he basically has a set list of contacts he goes through. It seems like almost all the developers of these systems know each other.
I remember being at a startup and being impressed that we were doing a million dynamic pages per hour. Then I got the chance to see a real system at a big company pumping out orders of magnitude more traffic than that per second. (I am not at liberty to share exact numbers.) And doing constant processing on that traffic.
I doubt your father would laugh at systems like that if he learned what they manage to handle.
Let's just say that any semi-reasonable back of the envelope will give you some truly impressive numbers. I don't think that I should say any more than that.
Now consider that this data is constantly being processed to figure out things like click through rates, trending requests, and much, much more. And that data is then fed back into the live site.
The implied transaction rate should be enough to impress the average mainframe user.
I don't actually think performance wise they have anything on a cluster of x86 servers. But its hard to benchmark things like that accurately.
From a developer point of view, its not a great technology stack to work on. Its old and out dated, and is pretty incompatible with other platforms. EBCDIC is a pain, 3270 terminal is a pain. Having to allocate files manually is a pain. Having actually about 7 different types of files is a pain.
Its really all a pain. Just dont go there.
Also, I found that guys who had been working on mainframes for a really long time have a pretty outdated view of what the "distributed" world has to offer. A lot of the time they dont understand what "web Scale" at the level of google, facebook et al are operating at.
As an additional funny annecdote, the I/O that a lot of people have been talking about is now conventional if high end disk array's AFAIK. But internally the disks are emulated for the OS to look like the disks it was using back in like 1975. Thats how it gets backwards compatibility, just dont change anything.
Can you imagine a time 1000 years into the future where software written today might still be in use? How cool would that be?!
It's a bit like the descendants thing that was discussed a few days ago on here - either lots of stuff will survive the next 1000 years (and therefore lots of running code will) or almost nothing will.
To name the ones I know of - vi, emacs, C, Common Lisp. These have held pretty steady for about 30 years now.
If banks still exist they'll still be emulating zOS on their quantum dimension-folded computers...
You hit it. Things break when you change things. The high priesthood that surrounds all mainframes prevents that from happening.
Nowadays, if you have to change stuff, IBM sells you AIX boxes that cost nearly as much as mainframes which sort of integrate with them to support those changes.
[1] https://en.wikipedia.org/wiki/Mainframe_computer#CharacteristicsIn any case, I'd advise anyone who can have a try at mainframe to give a look, even if just because it's one of the still used non-Unix based systems.
This one is worth listening to, for learning a bit about mainframes: http://www.se-radio.net/2012/03/episode-184-the-mainframe-wi...
Most companies are trying to stop using the mainframe so much, it is laughable though because they are moving from mainframe based solutions, to mainframe based solutions.
As someone who never has need to touch a mainframe, it's quite strange to look through this. It's not the 120 processor cores and 3 TB of RAM that surprise me; it's things like the cooling system and the I/O system that are so different from what I'm used to from working on a desktop system.
It's not exactly the computer that the world will only need five of, but focusing on machines like this is certainly closer to IBM's roots.
I've sometimes thought that a cloud provider backed onto z/VMs could be a nifty business model. Sure, you'd be priced above even the "expensive" providers like AWS; but then ... it's a mainframe. At the very least your customers would get hipster cred, which tends to attract a massive markup.
The IBM SmartCloud Enterprise+ Service (what amazing names IBM comes up with) does incorporate zSeries machines.
http://www-935.ibm.com/services/us/en/managed-cloud-hosting/
Mainframes are very, very extended nowadays. Researchers use them, the government uses them, some big business use them.
From my point of view, there are four "mainframe" concepts, and one of them is not a monolithic mainframe per se:
1. Supercomputers: thousands of CPUs, each with access to a few gigabytes of RAM, which can run massively parallel jobs
2. Shared memory machines: 64-256 CPUs, each of them can access to 0.5-2TB of memory
3. GPUs: Hundreds of CPUs, each with access to 1-4 GPUs and a few gigabytes of RAM, to run highly parallel vector jobs
4. Clusters: Commodity hardware (either racked or workstations) connected by a network which uses a queue system
The first three belong to the modern implementation of the mainframe, the last one is a "supercomputing"-like facility, built with regular hardware.
This brings us to OP's article, and to my main point. There are many research centres, business and government which use them, but there is a trend towards using regular Intel processors instead of mainframe-like vector processors.
Why? Because they're easier to program for. Us at the Barcelona Supercomputing Center have worked with ppc, ppc64, cell, ia64, i686 and x86_64 architectures, and here's what happened.
Some of these architectures don't have debugging/profiling tools available.
I joke with my colleagues about how it has been 10 years from the first CPU with hyperthreading and we still don't have fully automated tools to take advantage of parallelism on legacy code. We know it's difficult to do, but that has to be taken into account when planning to buy a new machine. Old code might run slower on the new machine because of the highly parallelism but lower raw speed.
Others (cell) are just so difficult to optimize that we needed 2 postdocs working 2 years only to make a matrix multiplication take about 80% of the full computing power. I'll only say that our next machine was going to be a Cell supercomputer, but that idea was dropped after a scientific panel advised against it.
Some months ago I took a course on GPU programming, and for those of you who have never programmed on vector processors, it is like the Hello World in brainfuck. My CS degree+masters wasn't just enough. It is overwhelming. The guy teaching the course was a PhD with 3 year experience programming GPUs, and he admitted that most of the times, if you compare the time it takes for you to program on a GPU with the time you save running the software, it is just not worth it. Hey, sometimes it is, and you save a lot of time. But not for most cases. So we're back to regular Intel processors again.
What I'm trying to say is that nowadays we buy machines for different purposes (large number of cores, large amount of memory, large throughput for vector multiplication, etc) but most of them are x86_64. GPUs are an exception because we buy software which has already been adapted to vector operations, but somebody had to spend many days adapting the code.
We work with Intel and IBM and we have access to the most recent CPUs and tools. IBM makes great hardware but they don't have anybody writing a development framework for that hardware, so you never reach even 70% of its peak throughput.
TL;DR: Yes, many companies use mainframes, but most of them try to put x86_64 processors in there, because they are easier to parallelise and there are many tested programming tools available.
Unfortunately, none of these are mainframes. Mainframes are a fairly different architecture and usually run OS/390 or z/OS these days. They really aren't about performance computing and are more oriented towards databases and IO.
These things are fascinating (as is AS/400, which is a mini and also an entirely different thing) in convergent evolution with modern hardware.
I'm not saying nobody uses them, but rather that there is a trend to use a NoSQL database on a regular Linux system rather than z/OS.
It's easier to find developers, more widespread support, and hopefully they will be more compatible in the future.
Edit: chuckMcM's comment [1] provides more insight on this. I must admit I've never worked with traditional mainframes myself, but I know of many research groups who are trying to develop new high-troughput data channels for commodity hardware (i.e. fibre-channel disks).
The current trend is going towards Intel+Nvidia+Linux. Anything else might be a good investment for some, but generally a bad idea for most.
What kinds of workloads though? I can see that "supercomputers" might be appropriate for modelling, analysis and reporting but that's a very different kind of scenario to highly available high throughput transaction processing that mainframes have traditionally excelled at.
You will be surprised, but it is due to the fact that banks were some of the first people to start using computing decades ago, and some of the programs I have seen in use at banks were older than me. Moving everything to a Intel based platform removes that backwards compatability, but z/OS is made with this in mind.
I don't know - for regular client-server computing, I've seen a mix at the 6 banks I've worked at - some Linux, some Windows Server, some Solaris. Most banks have a mix of all three, but have one that they "major" in.
Also, mainframes tend to be more of a Retail banking thing, for the core banking platform.Some Investment Banks use mainframes for back office stuff, but not many (in my experience).
Where I am working now, I have seen plaques in recognition for using a particular mainframe program from VISA for 20+ years. Also, the recently-retired core-banking platform from an acquired retail bank ran on mainframes, but was written in assembler.
Sorry, I don't mean to detract from your excellent points. I was just wondering if there was some miscommunication. The idea of me (a self-taught 24 year old) being more competent than a postdoc feels absurd, so it probably isn't true. Yet GPU programming is very easy for me.
Just an interesting mystery. I would like to help people out if I can.
If you understood this and still think it's easy, then congratulations you'll be able to pick and choose your jobs :)
(Obviously something like scrypt would be difficult to port to a GPU since by definition it's un-parallelizable, but other algorithms should be doable.)
The easiest way for me to solve this mystery is to try to port an example someone considers difficult. Do you know of any?
If you have a simple, straight through algorithm running on a few million datapoints. GPU's are great. If it deviates even a little from there, gpu's start to fumble. Writing those two algorithms is an afternoons exercise in Mathematica or matlab, but several weeks adventure on a GPU.
And remember computers are cheaper than humans (Currently, $0.12/hr at amazon and $10/hr minimum wage).
I'm serious. It's just so difficult for the majority of us. As relix said, if you find that easy, you'll make a lot of money working as a GPU programmer.
That being said, there are many real world problems where there are so many data dependences that transforming the input into independent vectors is plain impossible. For those cases, letting the CPU optimize the datapath is better than programming a GPU version yourself.
I really like the "RAIM" stuff that's apparently new. It seems like a similar idea not only to RAID, but also to this "lockstep execution" thing I just read about. I suppose the idea is that you can configure all or certain memory slots to be slaves such that all reads and writes are identical, or does this RAIM stuff also have RAIM with parity for reconstruction plus storage size increase?
Also, is the ability to install this thing on a normal (non-raised floor) pretty new, or has that been an optional feature of larger mainframes for a while?
Finally, the guy in the video mentioned some Java stuff. Do these things typically see lots of Java use or are they mostly programmed closer to the metal?
any other info?