Hacker News Reader

Comment by nickpsecurity | Hacker News Reader

OpenVMS or AS/400. Very common for admins to report systems running for 5+ years without crashes while doing real work. People occasionally forget where they're at and have to go looking for them. My company has an AS/400 employees never saw go down or get maintenance over 10 years. It's in use 24/7. They might have some fail-over setup or upgrade when nobody is using it due to traffic patterns. We've just never seen it fail. Their other benefit is they're very hands off with it nearly self-administering.

View on HN

Or that AS/400 calls the tech to fix it and s/he just does it while no one is around. Had a call once from IBM asking for directions since the AS/400 RAID cache battery needed replacing. No danger of losing data, the AS/400 was just running slower because it was bypassing the cache.

So... a tech would just show up at your office, ready to fix the machine? On the one hand, that is some fantastic service when you're running mission critical software. On the other hand, that must cost a whole lot of money and is quite devious of IBM.

Yeah, that type of service and phone-home is also common with $$$ enterprise storage (SANs). Funny story: in one instance I know of first-hand, the people running a NetApp filer setup with a redundant pair of systems had chosen to disable AutoSupport (NetApp's version of the phone-home support system where a tech would just show up with replacement disks when some failed) and take on support themselves. They did not test the alerting well enough after some network changes, which left the poor SAN unable to get distress signals through.

One day, the shared mount on a ton of systems around the company just stopped working. The master node (these have internal redundancy at the network, controller, shelf, and disk level) had failed over to the redundant node, which after time had so many disks fail without being replace that it went down hard. They had to work with NetApp support (at great expense) to get it back up, which took a few days. The system had been attempting to warn the admins as best it could of the impending doom for the better part of a year, and no one noticed.

The moral of this story is that you can have big-dollar redundant systems, but if you don't also have some dollars going to the care and feeding of them, it's wasted. The other lesson here is that such systems don't exist in isolation and are only as good as their weakest link, which in this case was a now-unreachable SMTP server. Lastly, although the LCD displays were trying their best to also signal this fault condition, people who would walk by these physical machines were used to this, so they didn't bother to actually read the errors. Do that at the peril of anything dependent on that system.

"The Fail-Safe Theorem: When a Fail-Safe system fails, it fails by failing to fail safe."

--https://en.wikipedia.org/wiki/Systemantics#System_failure

I love that book. Have you read https://www.amazon.com/Introduction-General-Systems-Thinking...? It's similar, but a bit more rigorous.

> The system had been attempting to warn the admins as best it could of the impending doom for the better part of a year, and no one noticed.

This is the area were inverted Unix philosophy applies: Give me good news as well. A weekly mail, that backups (or the filer or RAID or whatever) works A-OK. If the mail stops coming, something broke.

> The moral of this story is that you can have big-dollar redundant systems, but if you don't also have some dollars going to the care and feeding of them, it's wasted.

Agreed. Another moral: Don't believe anything works until you see it with your own eyes.

Had a friend unplug the UPS to test it while I was still configuring the new machine. "Can't really be sure unless you test it"

Especially true for backups. Make sure those aren't write-only!

Note to self, if need to socially engineer my way into the equipment room at a large company, pretend to be an IBM tech answering a phone home...

Good luck with that, as we do check these things before we allow folks to get to the servers. IBM has been doing this for a fair bit and we aren't exactly blind idiots.

It would be interesting where I am since you skip local crime and go straight to federal offense. Kept me away from booze in high school since its not minor in possessions, its federal bootlegging. I can only imagine what trying to enter a server room nets a person.

We just decommissioned a pair of EMC CX's. These things had phone home to EMC (and then eventually a third party maintainer) if something was about to go bad or had failed. We'd usually have an engineer onsite within four hours at the DC with all the right parts to carry out a fix.

By the time it was being looked after by the third party maintainer they'd just leave us a few disks and a couple of fibre channel controllers onsite and we'd get the DC remote hands to swap them out. The faulty parts were then boxed up and couriered back to the maintenance company all on their dime.

There was not an awful lot of involvement on our end except for authorising access to the DC when an engineer needed to attend.

None of this was massively expensive (and this was 24x7x365 cover) in the great scheme of things given their mission critical status.

> that must cost a whole lot of money and is quite devious of IBM.

You have essentially described IBM's business model. You pay more, but if something goes wrong someone shows up.

It actually isn't that bad on the budget. We don't have the same day service (I think its 24 - 48 hour response time). It sure beats the heck out of waiting for the disaster as the thing is pretty proactive. Much, much cheaper than service and support from Oracle, and Oracle doesn't visit or monitor the machine.

> So... a tech would just show up at your office, ready to fix the machine?

Yes. They are expensive and deviously good at it. Or, at least they used to be many years ago.

Many years back I worked in a tiny town (1000 people?) four hours from the closest depot and we even got same day service. This cost a king's ransom. But the AS/400 ran most of the business...

It's possible. I do want to try to get the uptime info from the people that run it if I ever see them. We regularly get techs come in to improve or fix IT systems. Just not that one.

I have had multiple years of uptime until I needed to do an OS upgrade for some new part of the financial system.

That's what vast majority of my feedback has said. Stays up until OS upgrade. Whereas, my research on VMS side indicated they do clusters that support rolling upgrades of the individual machines to not affect application. Might be around that remaining limitation of AS/400 uptime if they have software for that. Do they? Or are customers forced to upgrade to mainframe with Sysplex?

OpenVMS has an amazing cluster system, and yes they did upgrade one machine at a time in the cluster and lose no overall uptime.

I think the bigger IBM z machines do allow zero-time upgrades, but the iSeries (new name for AS/400) is basically a reboot the machine affair. I've never heard of Sysplex in regards to the iSeries.

They sent me two DVDs with 12 pages of instructions. The unnumbered first page is release notes and a notice that physical media deliveries would be an average of 5 to 9 business days on Jan 31, 2007 and beyond. (odd notice for a 2016 document)

Page 1 has my customer information with all sorts of tracking numbers. It also include the number of selected files (1,360), Kilo-bytes (sp) of data (3,963,529), fixes (3,394 fixes with 815 Kilo-bytes of data were superceded), etc.

Pre-install instructions finally show up on page 3 and the big warning about f'n it up shows on page 4. Page 6 brings us to the actual install which is written to be typed exactly with exact detailed results of each step.

to give an example

  2. Load the image catalog into the virtual optical 
     device using the following command:

  LODIMGCLG IMGCLG(ptfcatalog) DEV(OPTVRTxx) OPTION(*LOAD)

  3. Type GO PTF and press the Enter key

  4. Take menu option 8 and press Enter key

Took about an two hours counting the the 45 minute reboot. We have a very old and slow machine (need to buy a new one in the next two years, we are at a decade of use now) so reboots are a bit slow.

Do AS/400 and OpenVMS systems support live kernel patching? Can you patch the entire OS (even the lowest layers) without any reboot? If they don't, then you can't have zero downtime for years unless patches aren't being applied, which is rather bad for security.

With VMS you can hot add a machine to the cluster, migrate all running processes to it, then shut down the first node without any interruption. People kept systems running while moving buildings like this. And this was 20+ years ago.

Also supports cluster nodes with different CPU architectures. It was/is really slick.

That's something people keep forgetting that should be in any "Required Features for Good, Clustering Software" document. That they could move the software from VAX to Alpha to Itanium within same cluster was pretty amazing. Also a requirement for high-availability if a processor goes EOL. The other strategy is to bet on the market leader. Something they missed. ;)

Yeah, they (the new owner) is finally porting OpenVMS to x64. Their roadmap shows that this is going to be a long road, though. I can't remember their name...the company that bought VMS from HP. HP is licensing it from them for their own products.

It's this company:

https://www.vmssoftware.com/about_faq.html#faq1

They didn't buy it. HP is too greedy about the lucrative revenue & maybe patents they get off VMS that made me wonder why they killed it in the first place. Instead, it appears (not certain) that VMS Software Inc gets some OEM-like license that puts them in control of it while HP still owns it, can license it themselves (probably existing customers), and likely gets a portion of VMS Software Inc's licensing revenue. So, they've pushed off almost all responsibility for it onto another company without actually selling it or losing all the revenue.

Still a good development for VMS customers. Both legacy and people who will suffer through archaic stuff to reap benefits of bulletproof clusters. I'd be happy to. :) Only problem is it's basically got no security attention over the years with plenty of zero days or configuration issues lurking in there. If I deployed in a company, I plan to put network guards in front of it to absorb attacks while converting the traffic into something easy to process. Ideally, a PCI-based guard that forces it to only talk to the application's memory instead of rest of system. Plus safe-coded apps (Rust, Ada) and API wrappers to try to spot BS. That VMS supports cross-language development will probably help.

EDIT: The weird thing is they finished a port to Alpha ISA before they got to x86 ISA for migrations. I thought Alpha's already migrated to Itanium. Didn't see that coming lol.

Here's a guide on the main techniques it used for high-availability:

http://h41379.www4.hpe.com/openvms/whitepapers/high_avail.ht...

There remains the subtle distinction between downtime and unplanned downtime...

Most people recommend an IPL of an iSeries (AS/400) at least every 1-3 months. Did they just do that after hours?

Reply on news.ycombinator.com