Internet Archive Infrastructure
archive.org
archive.org
I found it all interesting. But the main takeaway for me is his response to Brewster Kahle’s question, beginning at 13:06, about why the IA does everything in-house rather than having its storage and processing hosted by, for example, AWS. His answer: lower cost, greater control, and greater confidence that their users are not being tracked.
And it is not only a cost comparison. You need different kind of people to manage in-house vs cloud, not that you need less or more, just different skills
Ideally of course they should be able to meet a fairly wide number of downstream eyeball ISPs at the major IX points in the bay area and offload a lot of traffic with settlement-free peering.
60Gbps outbound from AWS, Azure or GCP would be astronomically incredibly expensive.
Because of budget we tend to use things up 100% before putting more in.
It's been a huge blessing for a whole generation of startups to have a radically well-connected space that just about anyone can drop in equipment at multi-gig unmetered (albeit admittedly extremely constrained on power and cooling). It is honestly a part of what has made Silicon Valley great. Well, that and being able to cobble together a few replacement servers from Fry's components (RIP!) or schlep out to some ex-CTO's Sunnyvale garage in the middle of the night to offload some lightly used VA Linux 1U's...
Even today, you can get 15A and a 42U cabinet to call your own with unmetered gigabit for $400/mo - and probably less if you ask nicely.
That's not to say that it wouldn't be incredibly expensive, but probably far less than what you see on the pricing page.
I guess the biggest downside is the speed of scaling that they can do. As it is limited by how fast they can purchase and install new storage device. But with the use case of Internet Archive, that shouldn't matter much.
Also, bare-metal doesn't necessarily mean owning the hardware. You can rent it too. There are providers that provide bare-metal in one-click and sometimes available within minutes.
It really depends on what scale you're talking about. When you're a startup and suddenly land on the front page of HN, you might need 100x or 1000x your current capacity - in which case AWS will be useful to no end.
If, on the other hand, you're an established name with quite a bit of traffic already and the maximum uptick you will reasonably experience is 2x-3x, the argument holds far less water.
[0] http://radar.oreilly.com/2015/04/coming-full-circle-with-big...
[1] https://web.stanford.edu/class/ee204/Publications/Amazon-EE3...
I loved this and watched it to the end.
To any that feel the need to hide or disparage this because it seems to promote doing things on your own vs in the cloud, this isn’t some tech ops Total Money Makeover where you read a book and you’re suddenly in some sort of anti-credit cult. This is hard shit, and it’s the basics of the hard shit that I grew up with as being the only shit.
Yes, you can serve your own data. No one should fault you for doing that if you want. It takes the humble intelligence of the core team and everyone at IA to pull that off at this scale. If you don’t want to do the hard things, you could use the cloud. There are financial reasons also for one or the other, just as there are reasons people live with their family, rent, lease, and buy homes and office space- an imperfect analogy of course.
I hope that some of the others that could go on to work at the big guys or have been working there and want a challenge consider applying to IA when there’s an opening. They’ve done an incredible job, and I look forward to the cool things they accomplish in the future.
I am going to get a transcript done and up soon as well -- I just gave the talk on Friday so haven't had time to do so yet.
And that right there is why I continue to donate to IA. I am sick and tired of services offloading my data to destructive companies like Amazon.
For example are server logs considered tracking? It seems unreasonable to require that logs not be kept.
Edit: On further thought, I don't even know that tracking should be banned. Instead, I would argue that advertising in the way enabled by tracking should be banned. That way the incentive is removed with less bureaucracy.
We need to kill bureaucratic interests in EVERY THING.
Politicians should be conscripted servants with no method for empowering or financing themselves.
They should run on policy ALONE.
I don't agree there. There is no reason that someone has building bureaucracy as they're goal. The goal is to get something out of it, which is accomplished via the bureaucracy.
> We need to kill bureaucratic interests in EVERY THING.
While I don't necessarily disagree, this is orthogonal to the original issue.
> Politicians should be conscripted servants with no method for empowering or financing themselves.
I don't think conscripted means what you think it means...
Also, that would be impossible to implement. Either you don't try to cut off every path, in which case you have the option of limiting bureaucracy, or you try to cut off every path, increasing bureaucracy.
> They should run on policy ALONE.
I agree. Also orthogonal to the topic at hand. Also, please let me know if you find a way of implementing this without requiring bureaucracy as a critical component.
And yet, the opposite is happening. Government agencies are happy to tap into user data. And sometimes it is mandatory to keep data depending on what you are doing and your country legislation.
AFAIK the US is among the countries where you have the least such requirements, but there are still some sectors where logging is mandatory, like financial services. Many other first world countries (never mind dictatorships) require ISPs to keep data for a year or more.
Hate their "front end". One of the most disorganized user-facing sites.
I would love to see an effort to address that.
Would it be possible for archive.org to offer an API to allow other sites to present archive.org using their own front-end? We could see lots of specialty sites that focus on the user experience for some slice of archive.org.
Here is the entry point for API documentation: https://archive.org/services/docs/api/
Hot linking, CORS, and other things to support third-party integration are usually supported, though there are a lot of special cases for security or to prevent malicious use. If you run in to technical problems we are usually responsive to the main contact routes on the archive.org site.
The system is not designed to allow multi-party curation and editing of metadata, but there is nothing stopping folks from building third-party catalogs on top of the content stored (and served) from IA. That is sort of what openlibrary.org is for books. The same thing could be done for music, video, specific documents, etc.
Note: I work at IA but not on the APIs or archive.org collections
Most of content on IA loads pretty fast, so WB is a notable exception.
Wayback content is on the same disks as most other content, in the form of WARC files, with individual records fetched out of the middle of WARC files via HTTP range request.
Note: I work at IA but am not on core infrastructure team
Still, I strongly support the work they do and think it's very important work. I also think they do a good job for their size and resources.
http://web.archive.org/web/20120801000000*/http://blogs.msdn...
Why did you even make this comment?
I find it crass to gripe about Archive being slow. It's still much better than nothing. If folks want it to be faster than can donate rather than gripe, eh?
Good for you. Seriously (no joke, no sarcasm.) Good for you, that's awesome. (And yeah, that was the obvious question: "Have you tried giving them money?" but I wasn't feeling quite that salty.)
Well met. Sorry for being a grouch.
One aspect is that our data centers are in California, with no CDN. If you are on the other side of the world, you will have higher round-trip latency on every request, for all services.
Another is layers of caching. Popular or recently requested Wayback content is more likely to be in either an explicit cache (eg, redis), or implicitly in kernel page caches across all layers of the request.
Every wayback replay request hits several layers of index indexes (sorted by domain, path, and timestamp), which are huge and thus actually served from spinning disk over HTTP (!). This includes a timeline summary for the primary document, to display the banner. Then the actual raw records are fetched from another spinning disk over HTTP. This may result in one or more layers of internal redirect (additional fetches) if there was a "revisit" (identical HTTP body content, same URL, different timestamp). Then finally the record is re-written for replay (for HTML, CSS, Javascript, etc, unless the raw record was requested). Some pages will have many sub-resources, so this process is repeated many times, but that is the same as page load and you can see which resources are slow or not.
As mentioned in the video, depending on where we are in the network hardware upgrade lifecycle, sometimes outbound bandwidth is tight also, which slows down transfer.
And of course most of these services operate without a ton of overhead, so if there is a spike in traffic everything will slow down a bit. There is a lot of multi-tenancy-like situations also, so if there is a very popular zip file or Flash game getting served from the same storage disk as the WARC file holding a wayback resource, the replay for that specific resource will be slow due to disk I/O contention.
If you are curious about why a specific HTML wayback replay was slow, you can look in the source code of the re-written document and see some timing numbers.
Several organizations run large web archives that operate similarly to web.archive.org, and have described cost/benefit trade offs for different components. Eg, National Library of Australia has an alternative CDX index called OutbackCDX, which uses RocksDB on SSDs. I believe other folks store WARC files in S3 or S3-like object storage systems. The Wayback Machine is somewhat unique in the amount of (read) traffic it gets, the heterogeneity of archived content (from several crawlers, in older ARC as well as WARC), volume of live crawling ("save paper now" results show up pretty fast in the main site, which is black magic), running on "boring" general purpose hardware, and deep integration with our general purpose storage cluster.
Note: I work at IA but not on the Wayback system
Take for example this newer document: https://archive.org/details/manualzilla-id-5695071 The document has horrible name and useless tags, and the content seems to be only section 2 of some SW manual. How would I ever hope to find it if I needed that exact document?
Obviously such huge archive cannot be categorized and annotated by small team, so it would make sense to crowdsource the labeling process. Yet, as registered user I can only flag the item, or write a review. Why doesn't IA let users label content and build their own curated collections of items?
And I think they have bots that process the uploaded documents to do OCR and create previews.
This mature archived item https://archive.org/details/whattodrawhowtod00lutz/ is well described, but it is missing tags and it's not part of any relevant collection that would help us discover other similar books.
My experience with the website is that the original uploader can edit some part of the "metadata" (not all) of the uploaded document like title, description, topics/tags ..., but they can't move it to a different collection (initially the document is uploaded the "community" collection).
If they want to put it in a different collection they have to contact archive.org's staff.
Sometimes the staff notice the new files and move them to the correct collection or even create a new one for them (the latter case happened to me).
Are you getting "Fail with status: 498 No Reason Phrase"? You might have your Referer header disabled.
If that's the case, you can fix it by going to about:config and setting network.http.sendRefererHeader to 2 (or pressing the reset button to the right).
Don't know. I just spotted that it's the only meaningful difference I had in my traffic between visiting with Firefox and with Chromium. IIRC, it's the fetching of the capture timestamps that fails when the Referer header is missing.
> I haven't changed any settings from the Firefox default
You might have forgotten like I did once.
> and allowing it breaks links to jwz's site.
Do you mean this one?
Seems to be working fine for me with the Referer header enabled. Maybe it was a temporary glitch?
> The HTTP referer (a misspelling of referrer) is an optional HTTP header field that identifies the address of the webpage (i.e., the URI or IRI), which is linked to the resource being requested. By checking the referrer, the new webpage can see where the request originated.
https://tools.ietf.org/html/rfc7231#page-45
> The Referer header field allows servers to generate back-links to other resources for simple analytics, logging, optimized caching, etc. It also allows obsolete or mistyped links to be found for maintenance. Some servers use the Referer header field as a means of denying links from other sites (so-called "deep linking") or restricting cross-site request forgery (CSRF), but not all requests contain it.
Brewster Kahle does mention in the video (around 26:50) that one reason they use paired storage is so that the two disks in a pair can be in different countries. He then goes on to praise “Linux and the wonder of open source”; he recently blogged about that, too [2].
[1] http://blog.archive.org/2016/12/03/faqs-about-the-internet-a...
[2] http://blog.archive.org/2021/02/04/thank-you-ubuntu-and-linu...
This is preferred for its simplicity and performance, and if I were in their position I would do the same thing.
RAID 1 (mirrored drives) is similar to what the IA have described, but it sounds like they are creating their mirrors using simple file system commands (or tools like rsync) rather than introducing the complexity and overhead of hardware/software RAID.
Other forms of RAID (e.g. RAID 5/6) where data is spread across an array of drives with parity, would provide the IA with additional redundancy but at the expense of significantly increased cost and complexity.
Additionally, as I alluded to in the video, we regularly end up going down into the details and when this happens being able to closely examine and understand a disk's contents as written directly at the LBA using fibmaps and similar tooling is invaluable. I have personally been involved in the discovery of multiple possible-data-loss hard drive firmware bugs, and our catalog system is very paranoid about integrity checking. Modern hard drives are computers unto themselves and layering additional complexity on top of that (particularly complexity which might obscure or silently correct errors in the underlying datastream) is something I approach very carefully.
Perhaps you already covered this in the video and I missed it, but I was wondering how the team went about hardware upgrades/disposal? I maintain a just few servers in my own homelab so upgrading is pretty trivial for me, but I can't imagine what its like on that scale. Also, which hypervisor are you using to manage all of those VMs?
Thanks again for the talk, it was very insightful and fun to watch!
We try to keep a regular upgrade cycle (to hold to our tight budget), typically with a tick-tock of adding new hardware (expanding within our existing footprint) and cycling out old hardware and drives as they reach the far end of the works/doesn't work spectrum. We have a local partner who takes care of some disposal for us, but we also have no shortage of physical storage space, so we will also accumulate (sometime intentionally -- nearly our entire "red box" deployment -- see "previous version": https://archive.org/web/petabox.php -- is packed into a shipping container. We don't like to throw things away!).
For a hypervisor, we use Ganeti (running over KVM). Because our fleet is so heterogenous we need to be able to control a lot of VM parameters in order to efficiently pack our computational resources, and Ganeti is kind of in a sweet spot for us in terms of providing a lot more tooling than a bunch of virsh scripts, and being much smaller than systems like OpenStack geared towards large, homogenous deployments).
If they're willing to risk everything they've accomplished to date in order to issue a few extra copies of books, I don't see them surviving long term, nor do I feel comfortable donating to the project.
https://www.npr.org/2020/06/03/868861704/publishers-sue-inte...
Short of splitting a RAID array across two physical locations (a terrible idea), your proposal would require them to run mirrored RAID arrays in both locations.
This would give them greater redundancy, but would be a less efficient use of raw disk space than their current solution. It would also be more complex and difficult to maintain, and have performance impacts.
Have they lost data during rebuild?I know he briefly talked about that risk with different HD sizes.
Point is ceph & friends have a lot of overhead. Example in ceph: by default, a file in S3 layer is split into 4MB chunks, and each of those chunks is replicated or erasure-coded. Using the same erasure coding as wasabi,b2-cloud, which is 16+4=20 (or 17+3=20), each of those 4MB chunks is split into 20 shards of ~200KB each. Each of those shards ends up having ~512B to 4KB of metadata.
So from 10KB to 80KB of metadata for single 4MB chunk.
Note: seaweedfs doesn't actually support 16+4, it's set in 10+4 in source-code. But the architecture makes the low overhead possible.
In the video they mention storing everything in regular files on the filesystem. A regular filesystem would have inode overhead as well. XFS by default has 512 byte inodes (it can be more if you format it with bigger inodes, like you would for Ceph's Filestore backend).
For a lot of workloads Ceph's default erasure coding scheme (and Bluestore) would still be a lot more efficient than mirroring a file on top of a regular filesystem.
Yes that's correct, it's why Bluestore was created in the first place.