We might want to regularly keep track of how important each server is

Took me four reads to find an alternative way to read it other than "we asked some guy that doesn't even work for us to throw it on the ground repeatedly until the cover cracks open", like that Zoolander scene.

That's hilarious (probably for you as well, in hindsight). Do you feel comfortable naming and shaming this DC, so we know to avoid it?

I'd have considered calling a few friends from the fire brigade or the catastrophe protection there.

It's not an emergency, yes. However, if you want a situation for your trainees to figure out how to ventilate a building with the force of a thousand gasoline driven fans without anyone complaining and no danger to any person... well be my guest because I can't hear you anymore. Those really big fans are loud AF, seriously.

And, on a more serious note, you could show those blokes how a DC works. Where power goes, what components do, how to handle uncontrolled fire in areas. Would be a major benefit to the local fire fighters.

This was built into the building plan by room, with most rooms going down first and Meet-Me-Rooms + the rooms immediately adjacent where the big iron routers were, being the last to fail. It's been a while but IIRC there weren't any specific by-rack or by system protocols.

In a decentralized, self-service model, you can add "deal with convincing a stakeholder their app is anything less than most-critical."

Although it usually works itself out if higher-criticality imposes ongoing time commitments on them as well (aka stick).

Because not all environments are webapps which dozens or hundreds of systems configured in a cookie cutter matter. Plenty of IT environments have pets because plenty of IT environments are not Web Scale™. And plenty of IT environments have staff churn and legacy systems where knowledge can become murky (see reference about "archaeology").

An IMAP server is different than a web server is different than an NFS server, and there may also be inter-dependencies between them.

This sort of sequencing leads itself naturally to having a plan for limited capacity “keep the lights on” workload shedding when facing a situation like the OP.

Not everyone has elected to pay Bezos double the price for things they can handle themselves, and this is part of handling it.

(and here I remind myself again to write the screed against "cattle" metaphor...)

In heterogeneous system rooms, you can't stuff everything into a virtualization cluster with a shared storage and migrate things on the fly, thinking that every server (in hardware) is a cattle and you can just herd your VMs from host to host.

A SLURM cluster is easy. Shutdown all the nodes, controller will say "welp, no servers to run the workloads, will wait until servers come back", but storage systems are not that easy (ordering, controller dependencies, volume dependencies, service dependencies, etc.).

Also there are servers which can't be virtualized because they're hardware dependent, latency dependent, or just filling the server they are in, resource wise.

We also have some pet servers, and some cattle. We "pfft" to some servers and scramble for others due to various reasons. We know what server runs which service by the hostname, and never install pet servers without team's knowledge. So if something important goes down everyone at least can attend the OS or the hardware it's running on.

Even in a cloud environment, you can't move a VSwitch VM as you want, because you can't have the root of a fat SDN tree on every node. Even the most flexible infrastructure has firm parts to support that flexibility. It's impossible otherwise.

Lastly, not knowing which servers are important is a big no-no. We had "glycol everywhere" incidents and serious heatwaves, and all we say is, "we can't cool room down, scale down". Everybody shuts the servers they know they can, even if somebody from the team is on vacation.

Being a sysadmin is a team game.

Latency is important. Money makers > Latency sensitive > Optional requests > Background requests > Batch traffic.

Bootstrapping is important. If A depends on B, you might to drain A first, or A and B together.

The industry has this narrative because it suits their desire to sell higher-margined cloud services. However in the real world, especially in academia as cks is, the reality is that many workloads are still not suitable for the cloud.

I’m generally with you and the wider industry on the cattle-not-pets thing but there are a few things to keep in mind in the context of a university IT department that are different than what we regularly talk about here:

- budgets often work differently. You have a capex budget and your institution will exist long enough to fully depreciate the hardware they’ve bought you. They won’t be as happy to dramatically increase your opex.

- storage is the ultimate pet. In a university IT department you’re going to have people who need access to tons and tons of speedy storage both short-term and long-term.

I’m smiling a little bit thinking about a job 10 years ago who adopted the cattle-not-pets mentality. The IT department decided they were done with their pets, moved everything to a big vSphere cluster, and backed it by a giant RAID-5 array. There was a disk failure, but that’s ok, RAID-5 can handle that. And then the next day there was a second disk failure. Boom. Every single VM in the engineering department is gone including all of the data. It was all backed up to tape and slowly got restored but the blast radius was enormous.

And also importantly: you want to make sure you have a good enough description for them that you can say "terraform/cloudformation/ansible: make sure those are running" - without having to find them on the list and do it manually.

> proudly declares himself Unix herder

You know what herder's herd? Cattle. Not pets.

> wants to keep track of which systems are important.

I mean obviously? Industry does the same. Probably with automation, tools and tagging during provisioning.

The pet mentality is when you create a beautiful handcrafted spreadsheet showing which services run on the server named "Mjolnir" and which services run on the server named "Valinor". The cattle mentality is when you have the same in a distributed key-value database with UUIDs instead of fancyful server names.

Or the pet mentality is when you prefer to not shut down "Mjolnir" because it has more than 2 years of uptime or some other silly reason like that. (as opposed to not shutting it down because you know that you would loose more money that way than by risking it overheating and having to buy a new one.)

I guess that many companies that use "current practices" have plenty of services that they don't even know about running on their clusters.

The main difference is that instead of the kind of issues that the link talks about, you have those services running year after year, using resources, for the joy of the cloud companies.

This happens even at Google [1]:

"There are several remarkable aspects to this story. One is that running a Bigtable was so inconsequential to Google’s scale that it took 2 years before anyone even noticed it, and even then, only because the version was old. "

[1] https://steve-yegge.medium.com/dear-google-cloud-your-deprec...

Building your systems so that they don't depend on permanent infrastructure and snowflake configurations is an orthogonal concern from understanding how to shed load in a business-continuity crisis.

Well that’s more forward thinking than AT&T’s Nashville CO when it got bombed. They just depended on natgas grid for their generators if grid electric went out. They underestimated the correlation between energy grids.

When natgas got cutoff and the UPSs died, they had no ability to hook in roll-up generators and had to hastily install connection points (or hardline them). And no standby contracts for them.

I jumped up and down and convinced management to buy one of the first enterprise SSDs on the market. It was a PCIe card form factor and cost five digits for a tiny amount of storage.

We squeezed in the images using block-level deduplication and clever copy scripts that would run the compaction routine after each file was copied.

The difference was staggering. Just two of those cards made hundreds of other servers run like greased lightning. Boot times dropped to single digit seconds instead of minutes. Maintenance changes could be done at any time with zero impact on users. The whole cluster could be rebooted all at once with only a slight slowdown. Fun times.

I once had a system with layers of functionality; lvl 0 services were the most critical; lvl 3+ was "user shit" that could be sloughed off at need.

Had some stub servers at lvl 0 and 1 that did things like providing a file share of the same name as the lower level services, but not populated; so that inadvertent domain crossing dependencies weren't severe problems.

There was a "DB server" stub that only returned "no results." The actual DB server for those queries was on the monster big rack SPARC with the 3dz disks that took 10min to spin up fully. When it came up it took over.

Can you provide a cost centre or credit card for which they can bill this to? In case you didn't notice the domain, it is UToronto: academic departments aren't generally flush with cash.

Further, you have to have physical space to fit the extra cooling equipment and pipes: not always easy or possible to do in old university buildings.

Now which containers should your cluster stop running?

You haven't actually solved anything with this, you've just changed the abstraction layer at which you need to make decisions. Probably an improvement, but does not obviate some kind of load shedding heuristic.

If a redundant data center isn't an option, then you should put more into ensuring the system is resilient - fireproof room (if a server catches on fire it can't spread to the next - there are a lot of considerations here that I don't know about that you need to figure out), plenty of backup power, redundant HVAC, redundant connections to the internet - and you should brainstorm other things that I didn't think of.