Probably a good place to ask: is there a reason why Wayback Machine's archives often take forever to load?
Most of content on IA loads pretty fast, so WB is a notable exception.
Most of content on IA loads pretty fast, so WB is a notable exception.
Wayback content is on the same disks as most other content, in the form of WARC files, with individual records fetched out of the middle of WARC files via HTTP range request.
Note: I work at IA but am not on core infrastructure team
Still, I strongly support the work they do and think it's very important work. I also think they do a good job for their size and resources.
http://web.archive.org/web/20120801000000*/http://blogs.msdn...
Why did you even make this comment?
I find it crass to gripe about Archive being slow. It's still much better than nothing. If folks want it to be faster than can donate rather than gripe, eh?
Good for you. Seriously (no joke, no sarcasm.) Good for you, that's awesome. (And yeah, that was the obvious question: "Have you tried giving them money?" but I wasn't feeling quite that salty.)
Well met. Sorry for being a grouch.
One aspect is that our data centers are in California, with no CDN. If you are on the other side of the world, you will have higher round-trip latency on every request, for all services.
Another is layers of caching. Popular or recently requested Wayback content is more likely to be in either an explicit cache (eg, redis), or implicitly in kernel page caches across all layers of the request.
Every wayback replay request hits several layers of index indexes (sorted by domain, path, and timestamp), which are huge and thus actually served from spinning disk over HTTP (!). This includes a timeline summary for the primary document, to display the banner. Then the actual raw records are fetched from another spinning disk over HTTP. This may result in one or more layers of internal redirect (additional fetches) if there was a "revisit" (identical HTTP body content, same URL, different timestamp). Then finally the record is re-written for replay (for HTML, CSS, Javascript, etc, unless the raw record was requested). Some pages will have many sub-resources, so this process is repeated many times, but that is the same as page load and you can see which resources are slow or not.
As mentioned in the video, depending on where we are in the network hardware upgrade lifecycle, sometimes outbound bandwidth is tight also, which slows down transfer.
And of course most of these services operate without a ton of overhead, so if there is a spike in traffic everything will slow down a bit. There is a lot of multi-tenancy-like situations also, so if there is a very popular zip file or Flash game getting served from the same storage disk as the WARC file holding a wayback resource, the replay for that specific resource will be slow due to disk I/O contention.
If you are curious about why a specific HTML wayback replay was slow, you can look in the source code of the re-written document and see some timing numbers.
Several organizations run large web archives that operate similarly to web.archive.org, and have described cost/benefit trade offs for different components. Eg, National Library of Australia has an alternative CDX index called OutbackCDX, which uses RocksDB on SSDs. I believe other folks store WARC files in S3 or S3-like object storage systems. The Wayback Machine is somewhat unique in the amount of (read) traffic it gets, the heterogeneity of archived content (from several crawlers, in older ARC as well as WARC), volume of live crawling ("save paper now" results show up pretty fast in the main site, which is black magic), running on "boring" general purpose hardware, and deep integration with our general purpose storage cluster.
Note: I work at IA but not on the Wayback system