The IceCube Neutrino Detector at the South Pole Hits Paydirt

And never ever paste commands without first trying it into a trash terminal or editor.

A job has a task load that varies over time. Those tasks also have some maximum latency before Bad Things happen. Thus, you need enough worker capacity to handle the peak load. You can allocate new workers on demand, but that also has a certain warm-up time which can be problematic if it's lower than the maximum latency.

For most white-collar service-oriented jobs, the variance of that task load is quite smooth. If you're a software dev not working directly on a production service, there aren't that many urgent surprises. Also, the max latency is really high — many tasks can be put off till tomorrow, next sprint, etc.

That makes it easy to allocate just enough workers to handle your average task load, keep them busy all day, and rarely need to spin up or spin down new ones.

But some jobs just have really high variance or really low latency. No one knows when a patient is going to show up at a trauma hospital but when one does, you need the surgeon on site right now and not somewhere stuck in traffic.

For that kind of work, the logical thing to do then is to have spare idle worker capacity. It seems inefficient, but it's cheaper than the cost of missing your latency targets when a spike happens.

You can reduce the idle capacity needed by reducing variance. For example if you flew all your trauma patients to one hospital, that actually lets you allocate surgeons more efficiently. Because then at that point you have spikes frequently enough that the aggregate number of patients in a given day becomes more consistently predictable. Instead of one surgeon each sitting on their thumbs most of the day at ten city hospitals, you get three surgeons that reliably have a couple of patients a day at the regional hospital. (But of course there is the overhead of getting all the patients there.)

Also, you can reduce idle capacity by lowering warm-up time. That's why doctors have pagers — it reduces the time between a patient coming in and the doctor being ready to help.

Or you can increase the max latency, though this can be hard based on the nature of the work. For example, if you're better able to stabilize patients, you may be able to wait longer until the doctor is on-site.

In this case, the "install Debian" job has really high variance. 99% of the time it's a click, 1% of the time, you've gotta know some Linux internals. The max latency is low because you don't want the IceCube array non-operational for long. And the warm-up time is really high — fly someone out to the South Pole.

So the reasonable response is to throw money at idle capacity and send someone out pre-emptively.

Years ago, I interviewed for a couple of positions with Schlumberger which, at the time, basically went out to oil wells and made various measurements. In a lot of ways, it was essentially a tech job but the reason you had engineers (assisted by techs) was that you had various training and, presumably, other background for when things went sideways.

EDIT: I guess that is the reason why consultants come into the picture

Now walk across a carpet in leather-soled shoes. Do NOT even touch metal doorknobs, let alone metal faucets!

(since many of us spend most of our time writing code too...)

The water supply for the station comes from a rodwell, which is effectively a big bubble of molten ice, sustained by heat from the diesel generators. Once the rodwell gets to a certain size, some water can be extracted for use on station, and the bubble lowers progressively in to the ice. Eventually, the liquid surface in the well gets too deep for the pump in the well to pump water all the way to the surface, so a new well is started.

Waste water then goes in to the void left from the old well; poo, gunk off the dirty dishes, etc all included.