I'm trying to understand why the WARC format was made the way it is. Can someone explain the main reasons behind its design? Specifically, I'm curious about:
- Why not use a regular folder and file system with separate metadata files? It seems simpler and more user-friendly.
- Why not use SQLite?
1. Two reasons: First, many files are harder to manage. WARC files might contain hundreds or thousands of files. It's easier to manage big groups of files that are roughly the same size. Both for humans, and, at least in the past, for the file systems themselves. Second, once you break them up into files, what do you name the files? If you give them a name unrelated to the URL that was fetched, what is the advantage? If you name them based on the URL, suddenly you have a problem of mapping a URL to a legal file name, which can vary based on the file system. This would be a huge headache.
2. Yes, it predates SQLite, but also, why would you use sqlite? That's adding a huge amount of complexity. Is SQLite even good at storing big binary blobs?
Additionally, because of the clever way that WARC files are gzipped, each piece of the WARC file is gzipped individually, which allows random access into the file for reading enclosed content in a compressed file without needing to read the entire WARC file.
Nope! SQLite is good for lots of small-ish blobs (kilobytes), but once you start getting into the megabyte range, less so. There's also currently a hard upper limit blob size of 2GiB.
Filesystems don't generally deal well with having billions of files. Like you can work around some issues with deep directory structures, but even then it's not very effective and many cases you'll run out of inodes before you even reach a single billion. This is not a use-case filesystems in general tend to optimize for.
> - Why not use SQLite?
The main reason WARC looks like it does is to be as recoverable as possible if the crawler hard-crashes. There are much fewer things that can go wrong compared other formats. Records are only stored in exactly one location. Data corruption, write faults and other error states are all easy to detect and reason about. You have to work really hard to mess up a WARC file beyond being able to salvage most of it.
I disagree on the recovery part. SQLite being ACID compliant, arguably offers better recovery than the WARC format.
A CSV file is arguably harder to recover by comparison.
I just downloaded a sample WARC file to check for any checksums to detect bit rot, but I couldn't find one. Can you share any resources I can explore to understand how WARC is more resistant to hardware errors compared to other file formats?
First of all, no matter how easy it is to extract the files, the results will be kinda user unfriendly - unless someone goes through the archived html files updating all the URLs for images and javascript and so on, it'll probably end up looking pretty broken. For users who just want to view a few files, Wayback Machine is a better tool for the job.
With that said, some situations where warc is helpful include:
* If a website had a page at www.example.com/foo and a file at www.example.com/foo/bar.txt you can't express that in a regular filesystem, as you can't have a file and a directory with the same name.
* If a website uses some absurd 2000 character URL like https://s3.eu-west-2.amazonaws.com/document-api-images-live.... you don't end up with a filesystem-breaking filename.
* If www.example.com embeds an image from exampleusercontent.com you can capture the image in the same archive file.
* If for some reason you want to store daily copies of www.cnn.com in the same archive for comparison purposes - you can.
* It lets you store headers, 300 redirect messages, case-sensitive filenames, and all that sort of stuff.
* And it's an extremely simple format - basically human readable. So if you think you're archiving for the super-long-term and want to make really conservative choices, you can be pretty confident in plain text.