Senior Datacenter Network Infrastructure Engineer Full Time San Francisco, CA, US 30+ days ago Requisition ID: XXXX Salary Range: $140,000.00 To $200,000.00 Annually
- Capable
- Reliable
- Free
You going to be on call at 3am to bring services back up doing it for free?
Yes. For the Internet Archive I totally would. As responsible as I do my main job.
All it takes is one emergency and the volunteer person doesn't show up for the non profit to want to hire a full time person. This event has likely already occurred at Archive.
There's no reasonable world where this is a few hour's per week volunteering like your local boat club. It's a Top 200 website, around the size of the Guardian newspaper -- never mind that Wayback is only a small portion of what they do. (Book scanning, inter-library loans, paid services for other GLAM organizations and governments etc.)
The point is that someone needs to bring the service back up and they can't rely on volunteers to do it whenever they feel like it.
> someone needs to
Do they though? Why is there some assumed urgency? What happens if they don't do it "whenever they feel like it" (e.g. normal hours)?
Computers also generally don't randomly just stop working, and if they did in this case (network engineer), it's almost certainly the ISP anyway.
Maybe they want to hire someone full-time is so they don't have to have a policy of whenever they feel like it?
Having a reliable service is likely one of the reasons why their donors donate in the first place. Why do organizations and wealthy people still want to donate if the service goes down for 2 days and no one cares? Why would this kind of non-profit attract top volunteers in the first place?
People donate because they believe in the mission, and most of the money probably goes to the actual storage (making the cost of an employee probably pointless to care about, of course).
It would attract top volunteers in the first place because they'd think "I want this to exist, and perhaps they need some help." No one's going to think "wow the wayback machine was unavailable for two days? What a bunch of amateurs." Actually even if they paid me $200k, I'd assume they have some priority issues if they wanted me to be on-call and couldn't convince me of why that's necessary. That'd be a huge red flag for some nightmare mismanagement.
Depends on the service, but any effort to maintain this daft "gosh, what are they worried about, it's just the internet archive" line of thought is going to depend on either trivializing the needs of their staff, their users, or their computer systems. Good on them for taking things more seriously.
(3:00 AM on the west coast is work hours for some of their volunteers, users, and contributors across the globe)
> My library is closed at 3am every day!
A library will have IT staff on call. Staff need their computers to work when they come in in the morning, users need the infrastructure working in order to do pretty much anything - and why the hell shouldn't people be able to use an online borrowing service in the middle of the night?
Like I said in another comment, computers generally don't just randomly stop working, but no, a library worker doesn't need their computer at 9am so badly that they need an on-call IT worker (and presumably some kind of middle-of-the-night monitoring) to ensure it before they get in. Patrons don't have some incredibly urgent need to borrow a book such that we need 5 nines of uptime here. Or any SLA at all. It's fixed when it's fixed.
This attitude is how you waste a ton of money overcomplicating everything while making working conditions terrible for literally no reason. We should not "take seriously" things which are not serious just for fun. Nothing bad happens if the library computer is down for a couple hours beyond maybe some mild disappointment, or perhaps manually writing the data on cards like we did 30 years ago for a bit.
Hell, most banks in the US are "closed" for maintenance online for several hours every weekend (or at least they were as of a couple years ago) and no one cares. It's fine.
I work on systems that are in the critical path of 24/7 manufacturing lines, so I'm familiar with things that actually do rapidly cause problems if there's an outage. But that's not every computer system in the world.
And if they have people across the globe, then problem solved; they don't need a 3am on-call.
A single worker probably doesn't, yes. In contrast, hundreds of people being unable to do anything when they come to work in the morning is not a terrible reason to call someone and have them login for a few minutes.
Back to the IA, it seems to me our hundreds of thousands of users, employees and volunteers in time zones across the globe will be unable to access the service until we fix this is a decent reason for an institution that takes its own mission seriously to have an on-call. But you do you.
> Hell, most banks in the US are "closed" for maintenance online for several hours every weekend (or at least they were as of a couple years ago) and no one cares. It's fine.
(this is my industry) Some of their online services are closed, yes. The hours they'll be closed are typically posted ahead of time to minimize the inconvenience to everyone concerned and to display an understanding on the part of the institution that downtime is taken seriously and to signal it is not the result of carelessness. In contrast to online services, most institutions consider it a major failure if something like the ATM system is down for more than a few minutes.
> This attitude is how you waste a ton of money overcomplicating everything while making working conditions terrible for literally no reason.
That's remarkably backwards. The need for humans to be on-call is a result of systems being overcomplicated, not a cause.
If those users just say "oh, okay, I'll check back in a couple hours", then okay. Again, this is not hundreds of thousands of people without power or water or something. It's viewing or uploading to an archive. Like, if you're a user, do you really want someone woken up at 3am because you can't use the wayback machine right now? Because "they should take it seriously"? There's a person actually having to deal with that because... you really want to get that one Grateful Dead concert video from their free archive right now? I'm happy to say that yeah I hope their engineers aren't on-call after hours. I would much prefer that they get a good night's rest rather than fix it for me. I hope they're a cool place to work without delusional, psychopathic management.
Perhaps put another way, is there any computer service now that doesn't need a guarantee? Or is everything just critical now?
My point isn't that on-call is overcomplicating, it's that inventing and chasing an SLA often leads to expensive complicated systems (which, ironically, can often directly lead to lower uptime), and on-call has a human cost.
> and on-call has a human cost.
In the context of this thread, we’re talking about this job description:
Senior Datacenter Network Infrastructure Engineer Full Time San Francisco, CA, US 30+ days ago Requisition ID: XXXX Salary Range: $140,000.00 To $200,000.00 Annually
It’s not like we’re talking about indentured servitude or something.
> Perhaps put another way, is there any computer service now that doesn't need a guarantee? Or is everything just critical now?
That’s an interesting strawman you have there. I thought I killed it upthread when I wrote “Depends on the service.”
Misanthropic to me is making every IT job require getting up in the middle of the night to save a 100*epsilon of theoretical frustration lest we be perceived as lacking the requisite seriousness. And when someone points out that actually a given case is really not a big deal, we're met with "but hundreds of thousands of people might think 'oh, darn' for a moment!" And for a library, IME librarians tend to be really chill, so I suspect they'd, like me, attest that actually they'd prefer you not work at midnight on their account. Like I said, as a citizen funding and using my library, I sure hope they don't treat their employees that way.
At least half of a modern library's lending happens digitally. But let's put aside the example of the library.
> what is an example of a non-critical service in your mind?
I thought the example above of hundreds of people being unable to start their work day until someone solves an especially serious IT problem was universal, not unique to a library system's network. It's relevant to the Internet Archive, and even relevant to...
> Maybe the online checkin for a specific Great Clips?
Maybe not? Their entire WAN being down, affecting dozens of stores, possibly yes?
I'm not sure what's difficult here. Paying a person to be available to deal with a problem serious enough to justify the call makes sense. Sending trivial, less serious problems that can easily wait until business hours their way at night makes less sense.
Taking those calls isn't necessarily a bad job, but somehow that assumption is baked into what you're saying.
> Misanthropic to me is making every IT job require getting up in the middle of the night
Let's not do that, then.
I also wonder the cost differential.
Got me thinking about these incredible beautiful museums that are closed all night long… you’d still want someone there early enough to catch if the roof leaked so nobody missed their opening-hour tours. Back to libraries, bummer if I can’t renew my book online one night but would set reminder to try again later. Could affect my perception of my library’s tech prowess! That’s a risk for TIA! (Though I experience many a delay and downtime… perhaps my fault for using iCloud Private Relay like I bet scrapers sometimes abuse… but a few more hours overnight here and there wouldn’t be catastrophic for my current perception.)
Mainly would want overnight support to be cheap since stuff’s donation run. - Love TIA! An internet-billionaire should make history becoming a billionaire donor today (hope one’s reading this :) ).
> I wonder how many more hours of downtime The Archive would have experienced in 2025 if no one was ever woken up out of bed to fix it. (If anyone was)
Wasn't 2025 the year a kooky pro-Palestinian hacker collective decided IA was a good target for their idiocy? There was a great deal of overtime involved there.
I worked there (as a software developer) for 3 months and took a 75% pay-cut from my previous tech job. Luckily I don't live in SF! IIRC California employees were paid a little more, but not even remotely SV money.