That may have been all he did that time, but imagine if stuff had gone sideways, he'd have to troubleshoot it from Antarctica! I thought working from home was bad...
Protip: pressing up several times and then enter is very dangerous since the command that you will actually run may not have appeared on your screen yet.
I think the situation now is a lot better than it was, but a few years ago, most of the day there was no connectivity except Iridium. Most communications satellites are in orbits that don't take them near the poles, with Iridum being the obvious exception.
Related: https://en.wikipedia.org/wiki/GOES_3 was an amazing piece of the South Pole infrastructure for several years. There was a period every year where the Earth would eclipse GOES-3 - the batteries onboard were long dead - so the satellite would power down and everyone just sorta crossed their fingers and hoped it came back online with the right orientation and with enough electronics going so that we could keep using it. Bandwidth wasn't great, but it was quite reliable especially considering that it was launched in 1978.
And never ever paste commands without first trying it into a trash terminal or editor.
Despite (mostly) using vim as an editor, it's got to be emacs on the shell itself.
A job has a task load that varies over time. Those tasks also have some maximum latency before Bad Things happen. Thus, you need enough worker capacity to handle the peak load. You can allocate new workers on demand, but that also has a certain warm-up time which can be problematic if it's lower than the maximum latency.
For most white-collar service-oriented jobs, the variance of that task load is quite smooth. If you're a software dev not working directly on a production service, there aren't that many urgent surprises. Also, the max latency is really high — many tasks can be put off till tomorrow, next sprint, etc.
That makes it easy to allocate just enough workers to handle your average task load, keep them busy all day, and rarely need to spin up or spin down new ones.
But some jobs just have really high variance or really low latency. No one knows when a patient is going to show up at a trauma hospital but when one does, you need the surgeon on site right now and not somewhere stuck in traffic.
For that kind of work, the logical thing to do then is to have spare idle worker capacity. It seems inefficient, but it's cheaper than the cost of missing your latency targets when a spike happens.
You can reduce the idle capacity needed by reducing variance. For example if you flew all your trauma patients to one hospital, that actually lets you allocate surgeons more efficiently. Because then at that point you have spikes frequently enough that the aggregate number of patients in a given day becomes more consistently predictable. Instead of one surgeon each sitting on their thumbs most of the day at ten city hospitals, you get three surgeons that reliably have a couple of patients a day at the regional hospital. (But of course there is the overhead of getting all the patients there.)
Also, you can reduce idle capacity by lowering warm-up time. That's why doctors have pagers — it reduces the time between a patient coming in and the doctor being ready to help.
Or you can increase the max latency, though this can be hard based on the nature of the work. For example, if you're better able to stabilize patients, you may be able to wait longer until the doctor is on-site.
In this case, the "install Debian" job has really high variance. 99% of the time it's a click, 1% of the time, you've gotta know some Linux internals. The max latency is low because you don't want the IceCube array non-operational for long. And the warm-up time is really high — fly someone out to the South Pole.
So the reasonable response is to throw money at idle capacity and send someone out pre-emptively.
Years ago, I interviewed for a couple of positions with Schlumberger which, at the time, basically went out to oil wells and made various measurements. In a lot of ways, it was essentially a tech job but the reason you had engineers (assisted by techs) was that you had various training and, presumably, other background for when things went sideways.
Problem solved? Of course not. Employee safety is still a core KPI after all. So to address this problem, they installed a device in every company issued car which makes a BEEP! sound every time you drive aggressively (as judged by the device). After three BEEP!s you have to explain yourself to your boss.
I was extremely impressed with the safety-first culture when I interned at Schlumberger a few years back. Your coworkers will yell at you if you simply step into a work area with proper PPE (hardhat, safety boots, etc.).
EDIT: I guess that is the reason why consultants come into the picture
They had designed and built their own instruments, but they were on the plane so they could fix any problems, although the idea was to have no problems!
(I don't mean scientists are so dumb, but that they might have Antarctic Stare...)
They have some technical support at the SP but it's not great. In fact, one of my ex-coworkers, a well-respected research scientist, volunteered as a technical support person there (https://davidpablocohn.com/ and https://www.youtube.com/watch?v=eBaQtsft2bM). But I think he mostly helped people fix laptops.
Video chat isn't a realistic option - not the greatest connectivity down there...
Not sure how this could be the case. Dry air has higher thermal conductivity than moist air.
I imagine just circulating your nice toasty air into a larger area filled with moist humans who want to be warmer would work, but it depends on your building layout.
Now walk across a carpet in leather-soled shoes. Do NOT even touch metal doorknobs, let alone metal faucets!
The bigger problems with keeping ICL (the IceCube Lab - where the computers on top of the detector live, also the biggest datacenter on the continent) are somewhat more mundane though:
* If memory serves, there's one air handler that brings in outside air. Like anything else, it occasionally breaks or needs maintenance, sometimes in the middle of winter.
* Anything that has to operate in outside-ish conditions there, like the air intake for example, needs to be simple and robust. I think that early on in ICL's life, they had problems with frost clogging up the intake louvers, but that was before my time with the project.
* South Pole Station gets Cold, and so you need to be careful with mixing a little bit of outside air with much warmer inside air, to keep the heat evenly distributed and temperature from fluctuating too much.
* The station's HVAC is controlled by a DDC (https://en.wikipedia.org/wiki/Direct_digital_control) system, which is reputedly a pain to work on, and sometimes people get ideas about ways to improve it, which of course leads to new and interesting quirks in the system...
edit: formatting