Running a South Pole data center | Hacker News Reader

Running a South Pole data center

arstechnica.com

162 pointsdavidw58 comments

My grandfather visited Antarctica in '58-'59 as part of an initiative called something like "International Year of Science." After his second year at the University of Illinois Urbana-Champaign, he traveled to Virginia and hopped on a ship.

That ship eventually traversed the Panama Canal, crossed the equator, stopped over in New Zealand, then proceeded south to Antarctica. From there they then traveled north to Melbourne, then headed west. Apparently they had engine trouble and had to stop in Perth as well. From there the proceeded to stop in Durban and Montevideo before returning to Virginia. It was a long trip...

The kicker is that he had the foresight to buy film cameras (one color, one BW) before this and filmed a bunch of stuff. A couple years ago, while preparing for a move, my grandmother found a box with reels and reels of footage, unlabeled and not in any particular order. She took it to a film processing shop, and together with people there, they pieced together a probable timeline and spliced together ~90 minutes of interesting footage out of heap and digitized it. So now I have a DVD with some pretty cool footage of that trip. Elephants, albatross, icebergs, heavy machinery down in Antarctica... that must have been an amazing trip for a guy who grew up in small-town Illinois.

My family still has the coat and rucksack he was issued, with his name on it and a pretty cool seal/patch.

After that, my grandfather went on to do a bunch of other kickass things, like found a couple engineering startups in Boston in the 60s, with varying success: flip a couple, get screwed by some shady folks... the works.

Partly because of his story, I'm heading down to Antarctica early next year, though not on anything quite so cool: just taking a two-week cruise, but I'm excited to be retracing his steps in a way.

I would love to see this video. Would you and your family be willing to put it up on Vimeo, Youtube, or some video hosting (self-hosting too) service. Maybe edit it down to the highlights or something.

Or maybe since you're doing your own trip down there you can put together a video of your trip, spliced with your grandfather's trip.

Just some thoughts

I want to travel some more.

I've been meaning to do that and of course never got around to it. I plan to review the video before I head out there, so I'll be in a good position to put it online. Contact me via email (in profile) and I can send you a ping when I do.

I have also emailed requesting a ping. I imagine many people here would be curious to see it!

This is something that http://reddit.com/r/documentaries would like :)

Likely the International Geophysical Year http://en.wikipedia.org/wiki/International_Geophysical_Year#...

Yep, that's the one! I couldn't remember the exact name, so thanks for finding it.

Antarctica is personally very important to me and I've done a lot of research on it, there are some great old old videos made by the USG when they did a survey down there. I too hope to visit it some day, jealous you already have it planned!

But please please please upload the footage, I would love to watch it.

Opportunities like this provoke an incredibly intense response in me. I would drop almost everything to go work and live in Antarctica for 6-12 months, and searched fruitlessly for available jobs when I finished my undergraduate degree a few years ago. I'm not qualified for the position they talk about in this article, and I have a pretty good career path and professional responsibilities I can't just abandon now, but the pull remains strong.

Something about unique circumstance and desolation in particular just hits me in a particular part of my brain.

One of the most surreal and powerful moments in my life was pulling over in the middle of complete desolation in northwestern Namibia to wander the desert in the middle of the night. It was lit by a full moon, and the entire world was this shade of soft, ethereal purple I hadn't seen before and probably never will again. It was illuminating this tremendous expanse of landscape sparsely populated by utterly alien flora. In some ways it just felt right to be as utterly alone as I could be, in an alien situation I couldn't have imagined even 6 months prior.

A few years ago I saw this position opening and immediately applied for the same reason, it sounded like an incredible adventure. I was rejected as I should have been... unfortunately I'm a developer not an IT professional. Maybe someday they'll have an opening for an embedded developer and I'll get to spend a winter in Antarctica...

Edit: I looked through my old emails and I applied for the UNIX Systems Administrator position on April 25, 2012. I must have done it after reading this exact article!

Edit2: This is the original job description: https://web.archive.org/web/20120609030319/http://icecube.wi...

$52K/yr. That's really crap pay, especially when you consider that it's a contractor job with a 12-month option-to-renew.

Last time I looked into it you don't pay for room, board, food, cold weather gear, or the flight down/back. So in reality your take home pay and savings are fairly high.

Not everyone is a single male with no wife or kids. :-)

What's the tax situation in Antarctica? :P

The line "There is the possibility of travel to the geographic South Pole." makes up for it! Plus, I imagine that expenses are all (or mostly) paid for when in Antarctica, and the cost of living in Wisconsin is quite a bit lower then other parts of the US.

Jobs in Antarctica don't tend to pay particularly well. If you're doing a job like this, you tend not to be in it for the money.

A possible benefit is that you are probably not buying to many things while you are down there so it is mostly savings. Amazon shipping prices to the south pole are probably pretty steep even with Prime.

Yes, but cost of living must be incredibly cheap, maybe close to zero. I mean, what are you going to do with the money? You already live on site (rent is paid for), you eat food that is prepared at the research facility, and you can't exactly go out to bars on Saturday and buy women expensive drinks.

> and you can't exactly go out to bars on Saturday and buy women expensive drinks.

Alcoholism is a problem at Antartica.

http://www.funraniumlabs.com/2013/06/alcoholism-in-antarctic...

Your description of Namibia hits home. My family and I had the same sort of experience when we got stuck in the sand in very, very remote Saudi Arabia far from any paved roads or assistance. We were hopelessly stuck as the sun was setting and could do nothing but settle in for the night. But what a night. There were no lights or city glow horizon to horizon - not even an airplane overhead. Just a light breeze, blowing sand and the glorious stars overhead. Even though we were in a bit of a difficult and dangerous situation, I was utterly at peace.

If you're interested in work at the south pole, There are a few companies that have contacts down there, such as Scientific Research Corporation, and Lockheed Martin.

http://www.lockheedmartin.com/us/products/antarctic-support-...

http://www.scires.com/divisions/ice/index.html

I used to work at UW Space Science and Engineering Center, which did lots of remote support for the projects down on the ice. We would often be communicating with IceCube to plan shipments, or get machines that touched each other working. This was one of the most interesting systems support you can think of.

The hardest part of the support was system updates. There wasn't even a local yum cache on the continent, and many of these machines had security standards they had to meet involving staying patched. Once I managed to start a batch-update job over SSH, and within 15 minutes I had the team lead run into my office to see if I was doing anything, because he received a call saying we were saturating the bandwidth to the continent.

I never even considered applying for the jobs down on the ice, because I would never PQ. Even migraines can be too much, because if you're going down they need you ready to work EVERY day you're down there. Transient medical issues happen, but chronic without any control, or requiring medicine for survival, immediately disqualifies you. My friend who just hit McMurdo this week had to have dental work done to qualify.

All that being said, Barnett and the rest of the Ice Cube team do some amazing work with extremely difficult technical problems to solve. It is the same level of difficulty as anything in space, and they have done remarkably.

Very interesting. What does PQ mean? Pass qualification?

Physically qualify I believe, but either really works. Everyone would just grumble about all the health visits they would have to do in September and October to be ready.

I applied for this job. I have a PhD in physics and have some administrative experience, and I didn't (yet) have a postdoc position.

The application process was very interesting---I had a phone conference-call interview with a few people in Wisconsin and New Zealand. I made what must have been a relatively short short list, because the interview was not cheap: they flew me to WIPAC, had me take a full physical (ultrasound, cardio study, chest x-rays, etc.) which is required for any winter-over position, set up a full day of interviews with science-side, tech-side, and administrative-side people, some of whom themselves had wintered over. The beer and cheese curds in Madison are great. Check out the Great Dane.

A lot of the interview was to gauge personality, and there were a lot of questions along the lines of "If someone was acting in a dangerous & threatening way, and you were in a place more remote than the space station (in the sense that there is no escape capsule or any chance for rescue) what would you do?"

The tech that keeps IceCube going is a bunch of custom-designed and manufactured blades which receive (IIRC) UDP from the sensors in the ice. A lot of processing happens at the pole, because the limited (and satellite-orbit-dependent) bandwidth would make transferring the whole dataset wildly impractical. So the "interesting" events are found on-site and sent over the satellite, while everything is also written to tape. Once the summer comes the tape is swapped out. Scientists can also query additional data to be sent via satellite if they need something specific.

A few weeks later I was told that for the two IceCube winter-over tech positions, I was third choice, and that if one of the people offered the job sustains an injury, fails the psychological examination, or backs out, that I might be called on short notice. As I didn't have a job at that point, it was OK by me, but I was certainly disappointed. I'm happy in my postdoc position, but will certainly apply again when the time comes.

Had I wintered over, it would have been the smallest of small-world phenomena, as someone I know from college was one of the chefs there this winter.

Aside: I have heard that (if you get the job) they will sometimes preventatively remove your wisdom teeth / demand & provide other preventative treatments.

That's some crazy stuff. I wish you all the best when you re-apply!

For those interested in working at Antarctica, check out the Big Dead Place:

http://bigdeadplace.org/

From the old Antarctica FAQ on working there and how it changes you:

  First time is for the adventure, 
  Second time for the money, 
  And third because you can't work anywhere else

For those familiar with John Carpenter's The Thing, this is also an interesting read:

http://bigdeadplace.org/antarctica-the-thing-and-the-station...

  As a whole the subjects became less trusting and more
  suspicious of others immediately after their year in 
  Antarctica.” –A.J.W. Taylor, Professor of Clinical 
  Psychology, “The Selection of People for Work in Polar 
  Regions”

The essay ends:

  In the actual USAP, employees are forbidden flamethrowers.

My dad did something like this up in northern Canada at the DEW Line (http://en.wikipedia.org/wiki/Distant_Early_Warning_Line) to monitor the Russians during the cold war. He was a radio tech and while he told me stories about how bad the weather was combined with the loneliness, he also told me it was one of the most unique and best experiences in his life. It focused him into going back to school and becoming an aerospace engineer.

I went out for one of these roles back in the early 2000's. I even flew to Dallas on my own dime to attend a Raytheon career fair, in hopes of getting in the door at Raytheon Polar Services. The recruiters at the fair had little knowledge of these openings and I never got a call back. I had about 8 years *NIX system admin experience at the time.

Raytheon's contract expired and now support personnel are found through a variety of contractors. It's incredibly tough to get in the door at these places. I never got a single call-back from any of them back in the day, even though I was very competitive for jobs at private sector tech companies. I suspect that you have to have an inside contact or luck out and meet them at a career fair if you want to get your foot in the door.

Inside contacts definitely helped back when Raytheon had the contract. I can't imagine that's much different now.

I got a job working (labor) at Amundsen-Scott for a austral summer season when I was in college. I had three seasons of experience at a remote alaskan research station and I had several references from people who had worked down there. (There was a fair amount of crossover between the USAP and other NSF funded remote research stations)

Here's a list: http://ghgcorp.applicantpro.com/jobs/

You might also consider volunteering with a research group. Though they tend to select from pools of grad students, I would think that someone with technical skills (particularly hardware related) would be competitive.

Look at the job linked at the start of the article. There's always a few positions being floated either with IceCube, UW SSEC, or UW AOS that end up down there every year. That is also true of grad student positions, as you say.

If you're really interested in a job with IceCube, you can watch http://icecube.wisc.edu/jobs

We'll be looking for a couple more winter-overs in mid-January. The newest winter-overs landed at the South Pole less than 24 hours ago! You can follow their exploits at http://icecube.wisc.edu/news/current

Would any applicant from Europe be considered?

If you own an european passport I believe you could apply for a job at concordia, the most remote south pole station! It is 1100km inland in the coldest place on earth and 3233m above sea level!

Job board : http://www.institut-polaire.fr/ipev/l_institut/travailler_da... (job offers are pdfs)

More info : http://www.esa.int/Our_Activities/Human_Spaceflight/Concordi...

You may want to read about how to get there (if you know how to drive a truck you can apply to the heaby payload team that cross antartica once a year with haevy duty tractors, doing the travel from coast to concordia in 10 days) http://www.esa.int/Our_Activities/Human_Spaceflight/Concordi...

wikipedia : http://en.wikipedia.org/wiki/Concordia_Station

Yes, over the past few years we've had winter-overs from Sweden and Chile, and (as mentioned above) one of the outgoing WOs is Norwegian and one of the incoming WOs is German.

Yes, one of the two guys who just came back was Norwegian.

Twenty years ago, I spent ten weeks in Antarctica and the experience changed my life. It's hard to describe (and I'm not the guy to do it) but the bleakness and immensity of the continent is overwhelming--but then you realize that, hey, here are some bipedal mammals that flew in giant metal machines down here to eat, sleep, learn, and run IT systems. No big whoop, humans can do anything.

Every year, I still wistfully scan the Antarctica job listings...

I have a friend who did this sort of thing back in the '70s, maintaining a PDP-8 that ran a weather radar at the South Pole station during the long winter.

Not much in the way of creature comforts back then, he said they got very familiar with the few movies they had copies of, What's Up, Doc? was the only one he mentioned by name.

"An annual tradition is a back to back viewing of The Thing from Another World, The Thing (1982), and The Thing (2011) after the last flight has left for the winter."

http://en.wikipedia.org/wiki/Amundsen%E2%80%93Scott_South_Po...

That's awesome.

working in the south pole: very cool.

working for war profiteers: very uncool.

among other crimes against humanity, raytheon produce those illegal cluster munitions that indiscriminately kill kids long after the battle moves on. It is estimated that over 800,000 Raytheon cluster bomblets have been dropped on the civilian population of Iraq.

ckem.

This is an interesting line of thought.

I figure even a war profiteering corporation like Raytheon has some departments or projects that are a net positive for humanity.

By refusing to work at Raytheon, even in a positive role, you doom the company to hire less ethical people. if that happens enough, where good people won't work in good departments, eventually Raytheon will stop doing the good things, and become more evil.

alternatively, if smart people refuse to work for war, we might build a more peaceful future.

Just as an Example, Raytheon invented the microwave oven.

Something we could do without, and that doesn't make any murdered person alive again. I'll go even further and say someone who kills a person and then saves the lives of 10 is still a murderer. This stuff is not up for calculation, really. But even if it was: did they also invent microwaves, or do you think there is just a tiny chance that the microwave oven would have been invented either way, at some point or another?

Pretty sure every arms manufacturer is a war profiteer, and someone has to do it - doesn't mean they're bad people. They could have taken a pass on the cluster shit, though; that's just not necessary.

Why does someone "have to" do it? The only defense a military offers is against other militaries. So at best, soldiers defend against their own kind, at worst they maul civilians. It doesn't solve any problem, just the one it creates, and that only partially.

You're answering your own question. Any state without a military gets steamrolled by someone with. Now, in the nuclear and globalized world we live I'm currently, this may no longer be true, but you'll still be bullied around, at least.

I don't think along those lines. I don't see state A and their military on one side, and state B with their military on the other, and war profiteers somewhere else -- I see state A and B, their militaries and all war profiteers on one side, and civilians, the people who both get to die and pay for the stuff, on the other. E.g. the elites in China and the US have more in common with each other than with the populations they screw over, and those populations have more in common with each other than with their own so-called leaders.

Also, if a military is only useful to defend against other militaries, and that is the actual reason it exists, how did the first military organization come into existence? They had nothing to defend against, so maybe they just made something up? Oh look, the moon is so very round today, we need to attack the neighbouring tribe before they kill us all... I think it's perfectly possible that war has always been a racket, and that Stockholm Syndrome and nothing else keeps it going.

No counter-arguments, 3 downvotes? Whoever that was, to each their own. At least now you, even though it may not have occured to you yourself, cannot claim it has never been brought up in your presence.

I'm probably exactly the type of person they're looking for based on my background, and I've actually looked into going to Antarctica multiple times. I've always wanted to do at least a year long gig down there. The problem is that it's always contract-work, and I have never been at the right point in my life to give up a full-time job to drop everything and go.

I'm hoping if everything works out in my present organization, once we exit I can maybe take the time off needed to pursue an opportunity in Antarctica. That region of the world holds a particular draw for me because it's one of the least touched by humanity and has an almost ethereal natural beauty. It needs to be researched but also preserved and being a part of that would be awesome.

> the GOES-3 Satellite —a weather satellite launched in 1978 that lost its weather imaging capabilities and now provides 1-megabit per second data transmission for eight hours a day

Beautiful! From http://en.wikipedia.org/wiki/GOES_3

> having spent over thirty one years in operation, it is one of the oldest functioning satellites in orbit.

I worked with a guy a few years back that's done a few seasons with the Australian Antarctic Division. From everything I've heard it's an incredible experience.

For those who can work in Australia you can set up notifications for new positions (comms techs, electrical engineers etc) here: http://www.antarctica.gov.au/jobs/jobs-in-antarctica.

I just posted this Linux Journal article from 20 years ago, "Linux in Antarctica." We've come a long way.

[0]: http://www.linuxjournal.com/article/2843

[1]: https://news.ycombinator.com/item?id=8571947

They're hiring a webmaster. But I think it's in a different cold place: Madison, Wisconsin. http://icecube.wisc.edu/jobs/show/webmaster

Amundsen-Scott South Pole Station Antarctica Tour: http://youtu.be/P5lQ9DCXIbs. You can see the data center here for few seconds.

Hmmh smells like global warming..

Reply on news.ycombinator.com