Hacker News Reader

Very helpful to know that! Zimit[1] also uses warc files as an intermediate step to producing Zim files. You can use these Zim files to read and search websites offline with the excellent app Kiwix[2]. I think 'Kiwix for Android' and the Kiwix PWA support Zim files made with Zimit, with support by the desktop Kiwix application currently work-in-progress.

Other useful information about archiving websites is available from Webrecorder[3].

[1]: https://youzim.it/

[2]: https://kiwix.org/

[3]: https://webrecorder.net/

Are there any reasons why Kiwix doesn't use WARC/WACZ directly?

I think ZIM files contain way more data (like 80gb for Wikipedia) than a typical WARC, they're optimized for different things.

I've used it for about a year. ArchiveBox works well. My only complaint is that there's no JSON API to trigger archiving a page--it's primarily a CLI tool despite being built with Django. This can be worked around of course, and there's a browser plug-in to auto-archive sites you visit based on a whitelist.

API is coming soon (relatively, it's still a one-man project)! Stay tuned https://github.com/ArchiveBox/ArchiveBox/issues/496

I have an event-sourcing refactor in progress now to allow us to pluginize functionality like the API (similar to Home Assistant with a plugin app store), it will take a month or two. Next up is the REST API using the new plugin system.

Yeah, thanks for linking to the issue. I'm one of the upvotes on that. However, the age of that issue suggests to me that current contributors are (or have historically been) uninterested in a REST API.

Still, would be great to see such an update, and I'm still using ArchiveBox either way.

(Just read your bio, thanks for creating this)

It's not so much that the team isn't interested in building it, more that there there were some volunteers that indicated they started work on APIs already so I moved on to work on other areas of the codebase. Those draft PRs have since been closed as stale... so I'm planning to come back to doing the API myself in the next major version cycle.

You can also generate warc natively in iOS!

On a website, tap the share sheet icon, at the top of the tray tap “options”, and finally “web archive”. From there you can save/send/copy the file to your desired location.

Thanks, that's much better than iOS PDFs which don't preserve links.

I was positively stunned to learn such a thing, so while I don't have iOS I was able to find some references to that behavior in discussions.apple.com which led me to the fact that Safari on macOS can do it, too

But Apple gonna apple about using the plist hammer for every file format nail in their world, so no: definitely not the same thing we are discussing here

  $ xxd -l 512 'You can also generate warc natively in iOS! On a website, tap the share sheet ic... | Hacker News.webarchive'
  00000000: 6270 6c69 7374 3030 d201 0203 205f 100f  bplist00.... _..
  00000010: 5765 6253 7562 7265 736f 7572 6365 735f  WebSubresources_
  00000020: 100f 5765 624d 6169 6e52 6573 6f75 7263  ..WebMainResourc
  00000030: 65a5 040d 1216 1bd4 0506 0708 090a 0b0c  e...............
  00000040: 5f10 0f57 6562 5265 736f 7572 6365 4461  _..WebResourceDa
  00000050: 7461 5f10 1357 6562 5265 736f 7572 6365  ta_..WebResource
  00000060: 4d49 4d45 5479 7065 5f10 1357 6562 5265  MIMEType_..WebRe
  00000070: 736f 7572 6365 5265 7370 6f6e 7365 5e57  sourceResponse^W
  00000080: 6562 5265 736f 7572 6365 5552 4c4f 111c  ebResourceURLO..
  00000090: dd62 6f64 7920 207b 2066 6f6e 742d 6661  .body  { font-fa
  000000a0: 6d69 6c79 3a56 6572 6461 6e61 2c20 4765  mily:Verdana, Ge
  000000b0: 6e65 7661 2c20 7361 6e73 2d73 6572 6966  neva, sans-serif

  $ plutil -p - < 'You can also generate warc natively in iOS! On a website, tap the share sheet ic... | Hacker News.webarchive'
  {
    "WebMainResource" => {
      "WebResourceData" => {length = 5725, bytes = 0x3c68746d 6c206c61 6e673d22 656e2220 ... 3e3c2f68 746d6c3e }
      "WebResourceFrameName" => ""
      "WebResourceMIMEType" => "text/html"
      "WebResourceTextEncodingName" => "UTF-8"
      "WebResourceURL" => "https://news.ycombinator.com/item?id=39185542"
    }
    "WebSubresources" => [
      0 => {
        "WebResourceData" => {length = 7389, bytes = 0x626f6479 20207b20 666f6e74 2d66616d ... 64656e20 7d0a7d0a }
        "WebResourceMIMEType" => "text/css"
        "WebResourceResponse" => {length = 1880, bytes = 0x62706c69 73743030 d4010203 04050607 ... 00000000 0000065c }
        "WebResourceURL" => "https://news.ycombinator.com/news.css?15bzD7gVH5AOktnbu3y0"
      }
      1 => {
        "WebResourceData" => {length = 131, bytes = 0x3c737667 20686569 6768743d 22333222 ... 2f3e3c2f 7376673e }
        "WebResourceMIMEType" => "image/svg+xml"
        "WebResourceResponse" => {length = 1842, bytes = 0x62706c69 73743030 d4010203 04050607 ... 00000000 00000636 }
        "WebResourceURL" => "https://news.ycombinator.com/triangle.svg"
      }
      2 => {
        "WebResourceData" => {length = 315, bytes = 0x3c737667 20686569 6768743d 22313822 ... 2f3e3c2f 7376673e }
        "WebResourceMIMEType" => "image/svg+xml"
        "WebResourceResponse" => {length = 1842, bytes = 0x62706c69 73743030 d4010203 04050607 ... 00000000 00000636 }
        "WebResourceURL" => "https://news.ycombinator.com/y18.svg"
      }
      3 => {
        "WebResourceData" => {length = 43, bytes = 0x47494638 39610100 010080ff 00c0c0c0 ... 00000202 4401003b }
        "WebResourceMIMEType" => "image/gif"
        "WebResourceResponse" => {length = 1826, bytes = 0x62706c69 73743030 d4010203 04050607 ... 00000000 00000626 }
        "WebResourceURL" => "https://news.ycombinator.com/s.gif"
      }
      4 => {
        "WebResourceData" => {length = 5224, bytes = 0x66756e63 74696f6e 20242028 69642920 ... 636c6963 6b293b0a }
        "WebResourceMIMEType" => "application/javascript"
        "WebResourceResponse" => {length = 1945, bytes = 0x62706c69 73743030 d4010203 04050607 ... 00000000 0000069b }
        "WebResourceURL" => "https://news.ycombinator.com/hn.js?15bzD7gVH5AOktnbu3y0"
      }
    ]
  }

Wow I didn't know this, thank you!