VAXen, My Children, Just Don't Belong in Some Places
hactrn.net
hactrn.net
The modern comparison I would make is between the design of scale-neutral twelve-factor apps in language-platforms made for quickly spawning ephemeral processes ("scripting" languages like PHP, Ruby, Node, Lua, or even—in practice—Go)† and the design of "living system" VMs—and languages thereupon—that load and unload modules over time without shutting down (JVM app servers, Erlang, etc.)
It's okay to effectively "reboot" your Heroku instances to get them into a sane state. Not so with your databases, stateful load balancers, or VoIP servers. They come from two different worlds: one eschews state, while the other owns and manages it.
† Notice how all these languages are effectively wrappers around the same paradigm. There's a strong sense in which a "scripting" language is just a language that models the world in the Unixine fashion, such that it can be used to glue other Unixine programs together.
The only modern OS I know of that "relies on reboots for cleanup" is, ironically enough, the one famously descended from VMS (I am of course talking about NT-kernel-family Windows).
And this is not a VAX vs. Unix story; it's a story about by-the-numbers IBMers who dress in the Right Clothes and say the Right Things being completely feckless in the face of a VAX, development and maintenance of which has quite a different tradition and culture, mainly consisting of scruffy guys who Get Shit Done. Of which there is not a lack in the Unix tradition either.
A parable communicates a lesson, and the lesson "don't let ops people who have never run an architecture resembling yours near your machines" is kind of vacuous/trivial. Like I said, I'm looking for a piece of wisdom can be extracted from this story that's relevant for modern devops people.
I think the lesson is basically: some systems evolved around doing discrete batch-processing tasks that throw away their arenas when they're done, and have very few and discrete connection points; while other systems are "about" living object graphs that can't be disentangled, only upgraded as an entire system, and which maintain themselves rather than having maintenance done upon them from outside.
Rebooting solves problems with the former, since you're cleaning up all your temp files, cached intermediary results, etc. and then scheduling fresh new tasks. Rebooting the latter just makes it struggle to put itself back together exactly the way it was before. Think of power-cycling an LAMP "app-tier" server (Unixine) vs. a database server (Vaxine.)
To assure data integrity, an interactive database system has to log off all its users and shut down open databases before rebooting. If the reboot can't be orchestrated that way, if users can't be logged off in an orderly way, there will be data loss.
Yes -- all except the ongoing transactions at the users' workstations. That by itself could represent a serious synchronization problem. For a serious business, an emergency power system might represent a better choice.
I'm talking about Sybase here. But any database, which shifts the commit responsible to the operating system (without explicit guarantees that committed data is indeed written to disk) may invite you in for a huge and very unpleasant surprise.
It's about IBM mainframes (not Unix systems) on an uninterruptable ("immortal") power supply, a VAX which wasn't, but which could survive reboots in pretty good shape. Except when handling world-changing financial transactions.
Windows on the other hand still require reboots fairly regularly.
Note that around the time of this story, one of the most popular hardware platforms for running Unix (especially in academic environments) was ... the VAX (and its predecessor, the PDP-11). So in fact of the 2 "sides" in the story (IBM mainframes vs. VAXen running VMS), Unix would be on the scruffy VAX side (if anything, at that time Unix folks were scruffier than VMS folks).
Here's looking at you, ten-years-from-now tech website that will be sharing this with a whole new group of faces!
I've run across this story more than once. It's amusing but also more than a little troubling if that is in fact how fragile the global financial system is.
If however many billions of dollars were actually lost due to a server rebooting at the wrong time, would it actually be possible that it remained a secret, only documented in this one humorous and totally un-sourced account?
Also note the exaggerations of the incompetence of the IBM datacenter operators.
It's an entertaining read, because it's written to be an entertaining read ("a former gnome of Zurich"?). Maybe there's some kernel of truth in there; maybe not -- but there's no way to tell now (without other, better sources) what that may be.
And also introduced me to the units command which seems very handy!
Related are the faster-than-light stock-exchange trading orders of the past few years.
Some trades were originating out of Chicago for NY exchanges which, based on release timing of information, would have had to travel faster than light. Truth being that there's early access being offered some traders.
http://www.theverge.com/2013/10/3/4798542/whats-faster-than-...