Keep Our Servers Running
blog.archive.org
blog.archive.org
Here's just a few things we could use more experienced folks to help on:
- Solr performance, it currently gets quite slow under load and there are two read replicas but we think there may be some ways to improve it besides adding more replicas. You can run a full production copy of solr based on the dumps... But the current workflow to do this is kinda broke so I had AI fix it up but we haven't been able to verify if it's a fully working/good solution.
- We're doing big redesigns of major parts of the frontend. If your down to help for that Lokesh had many tickets open and guidelines and welcomes PRs.
- I'm currently migrating us from web.py to fastapi and could certainly use more experienced help for trickier ones.
Anyway, if you're actually willing to help all you gotta do is join our community call tomorrow at 9am PST and share your interests. We'll you on to something you like quickly :)
https://docs.openlibrary.org/everyone/community-call.html#co...
This is the first I hear of Litestar and I think it does seem very values aligned. I fastapi was picked simply because it was already known by the team and had a very thriving ecosystem. That being said, the hard part is getting off the old sync web.py stuff. Once everything is in async land moving to another framework if needed is relatively trivial. Thanks for sharing that though I'll keep an eye on it!
Was there any performance analysis done that indicated that the "code being sync" the main bottleneck?
Happy to help.
When traffic spikes, slow network calls to archive.org (for availability) and Solr tie up all available sync workers. The queue backs up, and suddenly every page becomes slow because there are no free workers left to handle requests.
This was handled from both sides: handling high-concurrency waiting better, while simultaneously optimizing Solr and removing archive.org from the critical path where possible.
1. Solr was used (I think still continue to be used) for many things it is not quite the right fit. It was done then because, it was simpler to do. For example, it is used to get list of editions when a work is loaded. Limit Solr only to search and figure out a better way to manage the other pieces, would be a good start. Search could be made a different service and the frontend could call it via API. Both of these together will remove Solr from the critical path.
2. The availability status of archive.org could come from a database and a backend task can continue to update it very frequently.
Lots of operational challenges come up when running a service for 14M patrons.
And Open Library in particular has a handful of challenges. 1. It's database has grown significantly (800+ GB) and Anand is right that IO (even on SSDs) is a challenge. The `thing` (infobase/infogami) triple-store design is well thought out and gets us a lot, and any system has to be tuned as it scales to hundreds of millions of rows. One strategy here is being smarter about cache and also shifting some of the load from psql to solr. Rishabh and others volunteers have been amazing assets as we've moved in this direction. Jim Champ on staff has been helping me tune psql, pgbouncer, and some of our high IO crons to improve raw db performance. 2. Limited hardware resources. We're trying to move some of our services within the Internet Archive's kubernetes cluster and we've done a great job migrating towards a world where everything is dockerized. It used to be a very painful process for our team of 3 to handle server ops, upgrades, and networking for nearly 15 manually orchestrated servers. One of the bare-metal racks running much of Open Library is significanly oversubscribed on vCPUs and so moving services off to free space and eliminate steal is critical for us right now. Our main web server (ol-www0) suffers from up to 20% steal and we're seeing a lot of congestion before requests even get to our web nodes (app servers). We have a plan and it takes time. 3. Open Library is still dependent on Archive.org for many lookups -- like book availability (which Ben Deitch has been helping me and Drini move into solr). When there are network issues and a network requests takes 5+ seconds, every web.py worker on that thread grinds to a halt and Ray's work moving us to FastAPI has made a significant impact 4. Solr. Drini has been heroic at restructuring our setup to use replicated solr in a way that has increased performance and relieved some of the pressure on our main cluster. This was a huge bottleneck for us this time last year and we've taken a lot of steps to ameliorate our situation. See: https://blog.openlibrary.org/2025/09/12/open-library-search-... 5. Raw spikes in traffic. We are seeing massive amounts of traffic that slams our book pages, increasing the pain of all the above. It saturates our limited resources, puts more strain on our database, ties us web workers... It makes modsecurity even more expensive. Part of the solutions is being more clever about provisioning, part of the solution is using fail2ban to prevent bad traffic from subtracting from the experience of the patrons who depend on us. Part of the solution is caching and optimizing our database to scale with load.
There isn't just one solution and the same 3 engineers on staff (and the support of a completely stellar community of dedicated volunteers fellows and leads) are doing our best to balance ops improvements with the necessary "product" and design improvements necessary that ensure we're useful to people to begin with.
I hope this gives the world a bit more of a glimpse how we operate and what some of our challenges are. We're an open source project and our goal is to share as many learnings as we can and to build something useful, sustainable, and beneficial for the community at large.
Thank you Ray, Anand, Drini, Jim, Lokesh, Lisa, Charles, and so many dozens more for your tremendous work (present and past) and thank you for being in our corner.
I'm all for ending copyright, but does any of that really need to be linked with the Wayback Machine? Huge chunks of the Web have been displaced or snuffed by social media and gen AI, making Wayback a unique(ly valuable) source of information and history. It's not my call, but it seems strange to risk it as a motte for the bailey of copyright abolition activism. What happens when some stick-up-the-ass company—whose stuff is probably in hundreds of different collections—decides it really wants a payday?
https://projects.propublica.org/nonprofits/organizations/943...
Rev. Costs(all) Legal
-----------------------------
2019 37 37 0.2
2020 22 20 0.4
2021 30 25 2.0
2022 31 26 1.8
2023 24 33 1.0
2024 27 24 2.2
It's happened:
- https://en.wikipedia.org/wiki/Hachette_v._Internet_Archive
- https://en.wikipedia.org/wiki/Universal_Music_Group_v._Inter...
I also hated that stunt. It undermined their mission, damaged all their goodwill and caused legal hassle. And all the while all these books were readily available to anyone who wanted them via Anna's etc.
They should have stayed on the right side of that line.
1. The state of access.
Tens of thousands of public schools and libraries in the United States and across the globe were suddenly mandated to shut their doors, leaving many students, parents, and teachers to fend for themselves, to learn from home, and unequipped with access to the physical resources many needed. Meanwhile, thousands of public libraries across the United States who had these resources were forced to closed their doors, meaning tens of millions of books the public had paid for became unavailable. Many of these public good organizations appealed for some way to solve the unique access challenges created by these unprecedented circumstances, which connects to point number 2.
2. Who was involved.
The details regarding how a decision is reached, and whether its made independently or unilaterally or with input and comment, are relevant when forming an opinion. In my experience, many people opine on this topic before researching this point. In 2020, more than 100 public institutions and libraries who were affected by COVID, joined in endorsing some temporary way to offer students, parents, and educators relief by connecting them with resources they had lost access to:
https://docs.google.com/document/d/1vkl3RX4CzpRTQsoG1tsdHC0f...
The reason many of us donate is for the internet archiving part of the business. When an organization starts taking donations and doing other things with them, including things that actively threaten the existence of the thing you were trying to support, it’s reasonable to stop donating.
> I'm all for ending copyright
downloading or streaming works for private use is permitted even when the source is illegal.
As book production goes digital and titles are increasingly sold as digital "leases" that customers and libraries don't own (i.e. kindle, libby, etc), it's becoming more and not less challenging (counter-intuitively) for libraries to archive and preserve books into the future.
And that's not a coincidence. And I wouldn't be surprised if we continue to see a steep uptick of similar challenges for archiving the web.
So, if you feel like you've been charged unfairly (e.g. product not what was expected), in error, fraudulently etc. then chargeback is the route and you should provide a reason. It's a mechanism to protect consumers. If on the other hand you just want your money back without good reason then chargebacks are not the right tool. The consumer also has an obligation to pay if there was no fault on the merchant side.
Honestly, sounds like everything is as it should be?
In both US and EU, chargebacks are a general consumer protection tool, implemented by card networks to make you less afraid of using them with potentially-untrustworthy merchants, even when not legally required.
They can be used for any kind of legitimate grievance, outright fraud being just one of them. If you book a hotel and the room is not up to the standards on their website, take pictures, take screenshots and chargeback.
They are not, however, a way to cancel a subscription you decide you no longer want (incidentally, neither is cancelling your card). Ask your lawyer, but this is actually illegal and considered fraud in many places. If you make a dumb choice, you have to live with the consequences or follow the contractual route.
This is unhinged. Then I can chargeback at McDonald’s because the burger doesn’t look the same as the picture. I still ate it and you probably still stayed in the room.
However I believe they have an easy cancel button if your donation is associated with an IA account, too.
There's one for subscriptions where you give it a monthly limit and if you want to end your subscription and they're being difficult you can just cancel the card.
It's great to basically never be beholden to anyone holding my payment info hostage. I know charge backs exist but while I've never had problems, I would imagine the more you end up having to go down that avenue the more likely you are to have problems using it.
> When you start a recurring donation of $25 or more in September, your initial gift will be matched 2:1.
They offer similar possibilities to what Visa and Mastercard offer, including changing your transaction amount post-factum (E.G. Uber everywhere, restaurants in the US for the weird way Americans do tip processing, some hotels for the minibar / phone / TV bill) and recurring charges.
They're not the same thing as in-app purchases, which are for digital content only, which Apple and Google charge much higher commissions on, and which have a much more pleasant cancellation / management UI. With IAP, Google / Apple is the merchant from your bank's perspective, this is not the case with Apple / Google Pay.
For example, I have a Pixel Care+ subscription, billed monthly. I can manage and cancel that subscription. Same for Google One, billed annually, and an in-app subscription (links out to Google Play). Can change payment methods, add or remove a backup method, etc.
So, if you did set up a recurring donation through Google Pay, go to payments.google.com (on a desktop system) and see whether it can be managed through the web interface there. You'll need to verify your account.
PayPal has also offered good tools to manage recurring payments and subscriptions. If you set up payments through PayPal then you should be able to modify or cancel them, without even involving the payee.
To keep favorable charity taxation status, a large donation can only be accepted if there are sufficient small donors who also contribute. It's complex and indirect, but the matching is quite real as a $100 contribution from a small donor enables the charity to accept a corresponding amount from large donor pledges.
Source: https://help.archive.org/help/how-do-i-make-a-wire-transfer/
SCT is not contingent on EU or Euro participation - SEPA membership is distinct from those. That is how Iceland, Norway and Switzerland can reap the benefits (the latter not even EEA member, though effective treaties are now somewhat similar). As long as an US bank has a correspondent branch/account/partnership/whatever inside SEPA, they can make it almost free to receive money via SEPA, and then possibly still save on currency conversion and transaction cost as they clear in bulk across the pond.
HOWEVER it's conceivable that the USA bank might end up charging some BS fees on their side just because they can. That's hard to control for.
You clipped it, but it's GB74RBOS... and the RBOS stands for Royal Bank Of Scotland.
https://en.wikipedia.org/wiki/Single_Euro_Payments_Area#Cove...
These types of schemes ring very poorly in my head for charity.
If someone has the ability to give 2$ for every 1$ I give, why are they only giving on contingence?
Just give the money!
If your mission is to help children with cancer, you don't want to spend 100% of your funding on chemotherapy and drug development, because that way, you eventually run out of money (donors die and their cards expire, and you aren't spending on marketing to bring in new ones). You may spend some of it on buying a fancy yacht and organizing a ball for the rich, which will bring in 4x as much in donations as you spend on getting it to happen, and that's the money you spend on chemo and new drugs. If you care about as much money being spend on cancer research as possible, that's potentially a better way to make it happen than spending your donations on cancer research.
Imagine a donor is willing to give you $1m, but you believe you can get up to $500k in extra donations from strangers on the internet if you do donation matching (it's essentially a free marketing campaign). So you ask them for $500k in hard cash and $500k in matched funds. Even if you overestimate and only get $400k in donations, you're still getting an extra $200k of free money.
Probably to avoid being shamed for not "just give the money". Exhibit A.
There are infinite ways to give away money, this scheme allows to align better with what causes other people are ready to support with their own money.
Both side get to feel like they got more out of the initial investment.
1:3 2:3
Everyone wins inside their heads and its nice to live life where everyone wins...
Also easier to justify charity when you think you are winning..
I feel like it's pretty good for what it is.
But otherwise you are right, this is done because it makes everyone donating feel like a winner other than few cynics like me. :D But that said I do not mind these drives.
These types of schemes are so common as to basically be universal. You'd probably be hard pressed to find a major charity that hasn't participated in something like this.
They'll ask me to round up my purchase to help feed the hungry. Why can't they not invest in all the infrastructure and systems to support this, and just donate a few nickels and quarters themselves. Plus they have the food to give.
I am not going to cry because the CEO cannot afford a yacht payment next month.
PayPal and Stripe are doing it too now.
0/ marketing
1/ credibility. Someone wealthy thinks the charity is important
If you publicly give $2m, people might think, that’s probably enough… no need to give more.
If you quietly give $2m usd and a tree falls in a forest…
If I say I’m going to match other people’s money 2:1 and that convinces people donate $90,000, my $180,000 yields $270,000 for the charity. I’d probably have written off the 200k anyway and just toss in the rest after, or throw it at another good cause.
Why? If i had the money, I would do the same.
If it is true, there is a value in getting people to donate something, however small, because it makes them more likely to donate or help in other ways in the future.
I've never actually got a 429 using a browser though personally, they do throttle aggressively though
Stop hiding public data and I might reconsider.
Internet Archive could buy Storj, probably for not much, keep 1 copy of their archive on the IA custom server setup they have now, keep the other copy on Storj, and free up half of the IA disk space. Storj automagically uploads data to 54 nodes, any 29 of which can reconstruct the original data (54/29 is where the 1.8 expansion factor occurs).
(and I am not saying it's bad, or indeed good, just that the usual expectations of being pro- or anti- cryptocurrency or AI don't easily map to things like the Archive)
https://news.ycombinator.com/item?id=29639222
> As noted the Internet Archive is experimenting with filecoin.io and storj.io and is always open to suggestions about how we might do our jobs better, and improve our service. We also host regular meetups (and have hosted summits and a camp) related to the Decentralized Web. See: https://blog.archive.org/tag/dweb/
Inside The Internet Archive's Infrastructure - https://news.ycombinator.com/item?id=46613324 - February 2026 (119 comments)
u/stavros proposed "Elephant" as well, which could be implemented today without Archive.org's explicit assistance.
Elephant system design - https://gist.github.com/skorokithakis/68984ef699437c5129660d... (A distributed, voluntary backup system (high-level design document))
https://news.ycombinator.com/item?id=46637992 (additional context)
(TLDR Every Archive.org item has a torrent, enabling distributed replication and serving of archived items and their contents, Wayback is a bit trickier; you can also ask your local nation state and museum/preservation apparatus to colocate some IA racks, if one is so inclined; The Internet Archive’s entire annual budget ($30M-$40M) covers 200PB storage plus all operations)
"Internet Archive could buy Storj"
Insanely convoluted solution that only someone with crypto-brain would ever come up with. You are really suggesting the Internet Archive should buy a bankrupted crypto company so they can store their data? What?
Also, even if you live in the least-expensive electricity places in the world, residentially it will cost you more in electricity to keep those platters spinning 24/7 [than the payout].
----
It is popular in the hobby cryptomining community to replace your under-desk spaceheater with a Monero mining rig (I also do this), and then only run it during wintertime (b/c it is effectively free mining, if you were already going to be running a resistive spaceheater).
Unfortunately, this cryptomining solution doesn't lend itself well to distributed-datastore cryptos because they are offline the majority of the year (i.e. files inaccessible).
----
As a part-time miner since 2012... I really wanted to like Storj, but if you want to get paid to replace your spaceheater at home, mine Monero on surplus CPUs (for reference: new $500 Ryzen798003xd mines at 5.5khashes/s; obsolete i5 still gets 3.8kH/s). Storj is in bankruptcy for fiscal reasons due to its impossible logistics.
Localized beneath my main office computer desk, this allows me to further reduce heating of my entire domicile.
----
Half of my kilohashing runs on obsolete i5 equipment (==free-to-me, minus geektime) – Monero Foundation strives to operate with GPU- & ASIC- –resistance; processors haven't gotten that much faster (in the past decade), and a surplus i5 isn't that much slower than a new $500 Ryzen (~2/3rds hashrate).
Monero's Windows7 client still works (fully supported), so perhaps you now have a use for your dusty-old laptop?
Should you need to upgrade the hard drive (mining currently requires full-blockchain, which is ~300GB and growing), you might as well experience linux (installing UbuntuOS is easy once you've created the USBmedia).
It is definitely not worth purchasing additional equipment, with residential rates, just to mine monero. I like warm toes, and have lots of spare CPU cores.
----
Reminder: this is only "free electricity" if you're offsetting RESISTIVE HEATING (i.e. not a heatpump). To actually reduce your powerbill and save the planet (the smarter choice), you should install a heatpump for wintertime heating.
This isn't a "guaranteed moneymaker," it's to heat your geeky underdesk toes – you are just making a fancier resistive spaceheater, which occassionally pays for itself.
You can usually get one for free (or less, "doing them a favor") via social media. I just gave one away (because the receiver had a better use for it than my XMR mining).
Make sure you use fan-boosting software, to keep everything less than 80°C. See you on the network!
[†] A MacPro5,1 with 12 cores will hash at about half the rate of a modern AMD Ryzen7 chip, using twice the electricity – perfect for your localized toe-heater (they also have incredible airflow, with MacFanControl software; in Linux, disconnect the PCI fan [∑] and the four CPU fans will full-blast). GPU is irrelevant for monero-mining (the stock GT120 [or 57X0] is perfect for this).
[∑] it's useless – or you can disconnect CPU_B_BOOST (fan), which is even more useless (but more-difficult/dangerous: do NOT disconnect _A_)... this makes the BIOS full-blast all remaining fans
NB: Newer i5s (mid-2010s) will hash at identical rates, but use much less power [not ideal for this situation] – but they tend to not have great heatsinks so you may have to run less cores.
Among the most important parts are having dual-channel RAM that is fast, and the G5 is going to not be capable of that.
https://blog.dshr.org/2026/01/internet-archives-storage.html
It's annoying how often archive.org blocks sites but they both complement each other very well.
https://liberapay.com/archiveis/donate
When finding archive.today's Monero donation address just now, I had to use archive.org because donatemonero.org [1] is down.
https://web.archive.org/web/20260526080016/https://donatemon...
Does it change the calculus at all if this "innocent" person was trying to dox the operator for nothing more than satisfying a personal whim?
https://en.wikipedia.org/wiki/Archive.today#2026_attack_on_G...
I don't think we should be supporting this random guy, no.
archive.today
archive.fo
archive.is
archive.li
archive.md
archive.ph
archive.vn
archiveiya74codqgiixo33q62qlrqtkgmcitqx5u2oeqnmn5bpcbiyd.onion
His identity must be secretive, as he has already drawn the ire of many powers that be, so… I personally think the ddos incident was justified… and I’m glad my bandwidth helped in any way, frankly. Some people seem to really want to portray him as some shady evil bad guy, and after all the hit jobs against him in the past year or two, I really wonder who these anti-archive.today people actually are and what they’re motivations are (besides money).
to DDOS one website, which is doxxing the owner. Not that it makes it good idea, but it's a little more understandable.
Oh and even if I wanted to ignore all of this, I couldn't use them anyways because they blackholed the majority of Finnish IP ranges after the DDoS incident to I guess punish the entire country for the targeted blogger being born here.
The media barrons in the west have bribed the US government to pursue archive criminally on their behalf for helping people bypass paywalls.
https://www.theverge.com/news/815691/fbi-subpoena-archive-is...
Is this really that hard to understand?
One guy blocking a single country is not at all problematic.
Edit: of course you should donate Monero
Everyone wants to scrape and extract from them 100% for free.
The donation got 3x, sure somebody is paying, right?
I was planning to donate, and this omission stopped me.
"Our matching donor wishes to remain anonymous, but we thank them for their generosity"
Simple.
It's also OK for primary donors to be suspicious of this for any reason that they choose, including no reason at all.
Maybe the matching-funds donor is all peachy and clean. Maybe they're the root of all evil.
We don't know.
This lack of knowledge doesn't let people make an informed decision, and people are allowed to seek to make informed decisions and to get turned off when that isn't possible. :)
(For the record: I'm more motivated to toss a few dollars in the hat when there's a multiplier at play than when there is not. I don't really care who it is that supplies the extra money.
My mentality might lead me to a situation wherein my donation to the local orphanage is anonymously multiplied by a nearby orphan crushing facility that turns lost kids into soylent green, but it's a risk that I'm willing to take.
Others are free to be as averse or accepting as they wish.)
The orphanage's rising success serves to limit the supply of orphans. This means that there aren't as many free-roaming orphans available for crushing as more of them get taken off of the market.
This improves the supply-and-demand ratio for orphan-derived products. A successful orphanage means that the the orphan crushing facility can make more money while doing less of that tedious work of -- you know -- crushing orphans.
(I don't know what kind of secondary or tertiary games could be played with anonymous matching donations to archive.org; my creativity has limits. But I do know that my support for enabling people to make informed personal decisions is approximately boundless, as anything less would be stifling.)
Personally, my worry about anonymous donation matching would be that the anonymous (to me) funder is influencing the recipient (to whom their identity is known), and without knowing who they are I can't know the nature of this influence or whether it should cause me to reconsider my support. But I don't see any extra reason to care about anonymous donation matching in particular, as distinct from any other funding whose source is opaque to me.
More money for the orphanage means more orphans can be taken in. That's good, right? But since the extra funds are not anonymous (to the orphanage), then it can come with influence on policies.
Policies like: All orphans are now accepted up to a certain age regardless of circumstance, and all orphans now get kicked out at age 12 regardless of ongoing need.
The stated reason is that the orphanage only has so many resources, and little kids need a lot more care than 12-year-olds do. So a line must be drawn somewhere, and that line is at age 12.
It's a popular policy: After all, when given binary choice, everyone would rather see a 3-year-old be taken good care of than a 12-year-old. Think of the little children!
For the orphan crushing facility that helped to impart this policy with their very generous policy-adjustment matching funds, that's a solid win: The orphanage now acts as a livestock kennel where orphans grow in size. There's still fewer of them coming out the far end, but they're bigger than they were before and are now much easier to process in bulk.
People know about the orphan crushing facility. They know they can't do anything about it; it just looms over there like any other dark fact of life. But the streets aren't full of roaming herds of abandoned kids anymore at all, so folks logically conclude that the 12-and-out policy must be working.
Out of sight, out of mind.
If they were the root of all evil then shouldn't you want to donate even more, to increase the amount of money they donate to the non-profit and reduce the amount they have left for their evil doings?
Evil has no boundaries.
Pseudo-profound statements on my HN?
Money going from the root of all evil to the Internet Archive?
Two cakes.
Also why arent they on opencollective? Surely thats a no brainer!
Senior Datacenter Network Infrastructure Engineer Full Time San Francisco, CA, US 30+ days ago Requisition ID: XXXX Salary Range: $140,000.00 To $200,000.00 Annually
- Capable
- Reliable
- Free
You going to be on call at 3am to bring services back up doing it for free?
Yes. For the Internet Archive I totally would. As responsible as I do my main job.
All it takes is one emergency and the volunteer person doesn't show up for the non profit to want to hire a full time person. This event has likely already occurred at Archive.
There's no reasonable world where this is a few hour's per week volunteering like your local boat club. It's a Top 200 website, around the size of the Guardian newspaper -- never mind that Wayback is only a small portion of what they do. (Book scanning, inter-library loans, paid services for other GLAM organizations and governments etc.)
The point is that someone needs to bring the service back up and they can't rely on volunteers to do it whenever they feel like it.
> someone needs to
Do they though? Why is there some assumed urgency? What happens if they don't do it "whenever they feel like it" (e.g. normal hours)?
Computers also generally don't randomly just stop working, and if they did in this case (network engineer), it's almost certainly the ISP anyway.
Maybe they want to hire someone full-time is so they don't have to have a policy of whenever they feel like it?
Having a reliable service is likely one of the reasons why their donors donate in the first place. Why do organizations and wealthy people still want to donate if the service goes down for 2 days and no one cares? Why would this kind of non-profit attract top volunteers in the first place?
People donate because they believe in the mission, and most of the money probably goes to the actual storage (making the cost of an employee probably pointless to care about, of course).
It would attract top volunteers in the first place because they'd think "I want this to exist, and perhaps they need some help." No one's going to think "wow the wayback machine was unavailable for two days? What a bunch of amateurs." Actually even if they paid me $200k, I'd assume they have some priority issues if they wanted me to be on-call and couldn't convince me of why that's necessary. That'd be a huge red flag for some nightmare mismanagement.
Depends on the service, but any effort to maintain this daft "gosh, what are they worried about, it's just the internet archive" line of thought is going to depend on either trivializing the needs of their staff, their users, or their computer systems. Good on them for taking things more seriously.
(3:00 AM on the west coast is work hours for some of their volunteers, users, and contributors across the globe)
> My library is closed at 3am every day!
A library will have IT staff on call. Staff need their computers to work when they come in in the morning, users need the infrastructure working in order to do pretty much anything - and why the hell shouldn't people be able to use an online borrowing service in the middle of the night?
Like I said in another comment, computers generally don't just randomly stop working, but no, a library worker doesn't need their computer at 9am so badly that they need an on-call IT worker (and presumably some kind of middle-of-the-night monitoring) to ensure it before they get in. Patrons don't have some incredibly urgent need to borrow a book such that we need 5 nines of uptime here. Or any SLA at all. It's fixed when it's fixed.
This attitude is how you waste a ton of money overcomplicating everything while making working conditions terrible for literally no reason. We should not "take seriously" things which are not serious just for fun. Nothing bad happens if the library computer is down for a couple hours beyond maybe some mild disappointment, or perhaps manually writing the data on cards like we did 30 years ago for a bit.
Hell, most banks in the US are "closed" for maintenance online for several hours every weekend (or at least they were as of a couple years ago) and no one cares. It's fine.
I work on systems that are in the critical path of 24/7 manufacturing lines, so I'm familiar with things that actually do rapidly cause problems if there's an outage. But that's not every computer system in the world.
And if they have people across the globe, then problem solved; they don't need a 3am on-call.
A single worker probably doesn't, yes. In contrast, hundreds of people being unable to do anything when they come to work in the morning is not a terrible reason to call someone and have them login for a few minutes.
Back to the IA, it seems to me our hundreds of thousands of users, employees and volunteers in time zones across the globe will be unable to access the service until we fix this is a decent reason for an institution that takes its own mission seriously to have an on-call. But you do you.
> Hell, most banks in the US are "closed" for maintenance online for several hours every weekend (or at least they were as of a couple years ago) and no one cares. It's fine.
(this is my industry) Some of their online services are closed, yes. The hours they'll be closed are typically posted ahead of time to minimize the inconvenience to everyone concerned and to display an understanding on the part of the institution that downtime is taken seriously and to signal it is not the result of carelessness. In contrast to online services, most institutions consider it a major failure if something like the ATM system is down for more than a few minutes.
> This attitude is how you waste a ton of money overcomplicating everything while making working conditions terrible for literally no reason.
That's remarkably backwards. The need for humans to be on-call is a result of systems being overcomplicated, not a cause.
If those users just say "oh, okay, I'll check back in a couple hours", then okay. Again, this is not hundreds of thousands of people without power or water or something. It's viewing or uploading to an archive. Like, if you're a user, do you really want someone woken up at 3am because you can't use the wayback machine right now? Because "they should take it seriously"? There's a person actually having to deal with that because... you really want to get that one Grateful Dead concert video from their free archive right now? I'm happy to say that yeah I hope their engineers aren't on-call after hours. I would much prefer that they get a good night's rest rather than fix it for me. I hope they're a cool place to work without delusional, psychopathic management.
Perhaps put another way, is there any computer service now that doesn't need a guarantee? Or is everything just critical now?
My point isn't that on-call is overcomplicating, it's that inventing and chasing an SLA often leads to expensive complicated systems (which, ironically, can often directly lead to lower uptime), and on-call has a human cost.
> and on-call has a human cost.
In the context of this thread, we’re talking about this job description:
Senior Datacenter Network Infrastructure Engineer Full Time San Francisco, CA, US 30+ days ago Requisition ID: XXXX Salary Range: $140,000.00 To $200,000.00 Annually
It’s not like we’re talking about indentured servitude or something.
> Perhaps put another way, is there any computer service now that doesn't need a guarantee? Or is everything just critical now?
That’s an interesting strawman you have there. I thought I killed it upthread when I wrote “Depends on the service.”
Misanthropic to me is making every IT job require getting up in the middle of the night to save a 100*epsilon of theoretical frustration lest we be perceived as lacking the requisite seriousness. And when someone points out that actually a given case is really not a big deal, we're met with "but hundreds of thousands of people might think 'oh, darn' for a moment!" And for a library, IME librarians tend to be really chill, so I suspect they'd, like me, attest that actually they'd prefer you not work at midnight on their account. Like I said, as a citizen funding and using my library, I sure hope they don't treat their employees that way.
At least half of a modern library's lending happens digitally. But let's put aside the example of the library.
> what is an example of a non-critical service in your mind?
I thought the example above of hundreds of people being unable to start their work day until someone solves an especially serious IT problem was universal, not unique to a library system's network. It's relevant to the Internet Archive, and even relevant to...
> Maybe the online checkin for a specific Great Clips?
Maybe not? Their entire WAN being down, affecting dozens of stores, possibly yes?
I'm not sure what's difficult here. Paying a person to be available to deal with a problem serious enough to justify the call makes sense. Sending trivial, less serious problems that can easily wait until business hours their way at night makes less sense.
Taking those calls isn't necessarily a bad job, but somehow that assumption is baked into what you're saying.
> Misanthropic to me is making every IT job require getting up in the middle of the night
Let's not do that, then.
I also wonder the cost differential.
Got me thinking about these incredible beautiful museums that are closed all night long… you’d still want someone there early enough to catch if the roof leaked so nobody missed their opening-hour tours. Back to libraries, bummer if I can’t renew my book online one night but would set reminder to try again later. Could affect my perception of my library’s tech prowess! That’s a risk for TIA! (Though I experience many a delay and downtime… perhaps my fault for using iCloud Private Relay like I bet scrapers sometimes abuse… but a few more hours overnight here and there wouldn’t be catastrophic for my current perception.)
Mainly would want overnight support to be cheap since stuff’s donation run. - Love TIA! An internet-billionaire should make history becoming a billionaire donor today (hope one’s reading this :) ).
> I wonder how many more hours of downtime The Archive would have experienced in 2025 if no one was ever woken up out of bed to fix it. (If anyone was)
Wasn't 2025 the year a kooky pro-Palestinian hacker collective decided IA was a good target for their idiocy? There was a great deal of overtime involved there.
I worked there (as a software developer) for 3 months and took a 75% pay-cut from my previous tech job. Luckily I don't live in SF! IIRC California employees were paid a little more, but not even remotely SV money.
I would really appreciate it if they spent more resources on improving the site’s UX and fewer on picking pointless and expensive copyright fights.
https://bsky.app/profile/textfiles.com/post/3musdwx6kqk2c
Fundraising drive coordinated with their loudest employee saying something else, because... of course.
Set up after the lawsuits that nearly wiped out his personal assets, Brewster Kahle's latest foundation shows $743 million in assets.
https://projects.propublica.org/nonprofits/organizations/991...
“We survived,” Internet Archive founder Brewster Kahle told Ars. “But it wiped out the Library.” https://arstechnica.com/tech-policy/2025/11/the-internet-arc...
"Do you do backups too, for example to guard against corrupt data getting mirrored across both copies, or accidental deletion?"
John Gonzalez, Internet Archive infrastructure lead: "We have done experiments to confirm that we can back up large portions of our corpus... but this is not a regular practice for us at this time."
https://blog.archive.org/2016/10/25/20000-hard-drives-on-a-m...
Depending on when you click his feed, the ongoing downtime is due to DDoS attacks, bad actors, AI scrapers, power problems, or now an inability to source hard drives.
Does that occur? Who's doing the matching?
I get that many people like such an archive to be around for consumption. Me too. But, keeping emotions aside, I can't find a justification to violate the rights of creators.
I'm sure many people here will have the knee-jerk reaction to immediately downvote this question. But I know there are also thoughtful people on HN. If some of you have ideas about the legal and moral justification for archive.org's actions, I would like to hear them.
Perhaps it is time to recognise that this literal Mickey Mouse law needs to be rebalanced more in favour of the public than the monopolist, as that creates better societal outcomes.
Any creator-based outrage should be directed first and foremost at the for-profit violations by AI companies, rather than the small volunteer service which is keeping history up.
Ultimately there is no archiving without piracy. The alternative is for the web to exist in an eternal present where everything vanishes the moment some business decides it's not worth it or conflicts with current corporate values.
Because copyright isn't black and white like that. Its a lot more complicated. Its likely a lot of what internet archive is doing is fair use (not all though, they do push the envelope, e.g. their covid library stunt).
As for morally, while that is going to depend on what moral system you subscribe to. Copyright is a relatively recent invention. The bible might say thou shall not steal, but it's likely in context that the bible was fine with copyright infringement. It probably not even something that would have been thought about in a bronze age context.
As far as utilitarianism goes, at first glance seems like Internet archive increases happiness more than it decreases it, so that is a check.
I dont see any reason to think the internet archive violates the Categorical imperative either.
so i guess the question is, under which basis do you believe it to be immoral.
Depends on ones morals, of course.
Shut ClimateGPT down, apologize and then we can talk about donation.
Brewster Kahle, Internet Archive founder (and board chair), is on the board of Erasmus.AI.
It does seem weird to rattle the donation box because hosting costs are going up while shooting money at the prime cause of hosting costs going up.
I'm sure they have separate bank accounts, but let's not pretend these organizations are independent. They're all run by the same little club of friends.
and it's darkly funny how HN downvotes this when every single answer on Mastodon is incredibly negative
Mastodon is full of, for the lack of a better word, nuts. Every time I've had the displeasurable experience of clinking a link and seeing that dark violet colour cover my screen I know I'm in for either adults who never matured past their early teenage years or ideological hyperpartisanism, and within five seconds my prejudices are confirmed. Social media doesn't magically fill with rational people the moment you "fix" the infrastructure.
https://mastodon.archive.org/@internetarchive/11721950124572...
Actual modern day luddites, it's going to be so hilarious in a couple years from now when they all stop caring about AI and move on to the next 'current thing'.