Hacker News Reader

Comment by jedieaston | Hacker News Reader

Remember: there are BitTorrent links that the Wikimedia Foundation gives out of SQL dumps of Wikipedia and the other projects. You can have a copy in case this happens in your country: https://en.wikipedia.org/wiki/Wikipedia:Database_download#Wh...

Also, the Kiwix project has a hotspot project that allows you to host ZIM files (dumps of Wikipedia and other CC licensed content, like TED talks and StackOverflow) on a Raspberry Pi, allowing you to share it with others. Setup info here: https://www.kiwix.org/en/downloads/kiwix-hotspot/

View on HN

There's also a read-only IPFS mirror of Wikipedia in English: https://ipfs.io/ipfs/QmXoypizjW3WknFiJnKLwHCnL72vedxjQkDDP1m...

I love ipfs. Can this actually be ddos’d as well?

The ipfs.io is just a web-based way to access IPFS, called a gateway. There are a bunch of different gateways. In addition, you can run an IPFS node locally, and then as long as just one node holds the content you're looking for you are looking for you're good. There are also browser extensions to re-write gateway URIs to localhost URIs.

Does this actually answer the question? If there's a node online that means I can reach the content, but would it help with DDOS? Not so sure.

Ipfs rarely stores 100% of content on one node.

A popular IPFS file might be available on thousands of nodes similar to how popular torrents have thousands of seeders. DDoS attack against thousands of servers across multiple countries and networks would be nearly impossible to perform.

It depends on the kind of node storing the data and how many there are. It's likely easier to DDoS 100 people on DSL than a single Wikimedia endpoint.

So there is no central place like bittorent tracker which if down the network does not works?

Or is it like DHT which does not need central tracker?

IPFS uses DHT on a fairly fundamental level.

Yes. It's based on a bittorrent-like protocol with a DHT, so you can get a list of nodes hosting a particular file; then you can DoS them.

Caveat: the last full Kiwix English Wikipedia archive was made in 2018. They could use some help with automating their build process if anyone here has the time.

From a cursory glance at the site and source code, it's really hard to see who/what is involved with building an archive. There's automated builds set up for the Pi image itself.

Agreed. I can see that other Wikipedia languages are crawled - https://wiki.kiwix.org/wiki/Content_in_all_languages shows dozens of updates this week - but the best leads I have involve poking around the openZIM Github org, https://github.com/openzim . There might be a running "zimfarm" somewhere?

Looks like you are right, you may be able to join the farm to help ( i have not tested as I am away from my computer at the moment )

https://github.com/openzim/zimfarm/blob/master/worker/README...

Someone should ask drone.io or packet.net for an Epyc machine to do this.

You can build your own ZIMs from any MediaWiki instance using this tool: https://github.com/openzim/mwoffliner.

Maybe it would be worth putting together another zimfarm that is constantly updating.

The last time I checked, it was more a problem of lacking servers with sufficient resources: https://phabricator.wikimedia.org/T124960 https://phabricator.wikimedia.org/T219078

It sure doesn't harm if someone creates their own ZIM files and reports on their results (and/or shares the resulting files).

I'd actually love to see a fully working IPFS fallback for wikipedia when regular hosting doesn't work. Would it even be possible with ipfs?

IPFS has a Wikipedia mirror but it is fairly out of date since it is dependent on the Kiwix archive.

https://github.com/ipfs/distributed-wikipedia-mirror

From a comment above: https://ipfs.io/ipfs/QmXoypizjW3WknFiJnKLwHCnL72vedxjQkDDP1m...

Assuming that all formats contain the exact same data, i.e. they were generated at the exact same time, which is the (1) most useful for offline viewing (2) most future proof for archival and backup? Is there another, more viable/useful format?

The XML dumps are the most compact and sustainable format in the mid term (let's say decades). https://meta.wikimedia.org/wiki/Mirroring_Wikimedia_project_...

ZIM might be able to survive longer (centuries?) as probably the future will still need some HTML parser, while wikitext parsers or PHP might be long dead, who knows.

Thank you for the answer. So especially for personal use I am better off hoarding the ZIM version, especially considering there is the dedicated Kiwix reader, while I am not aware of a similar tool for the XML dumps.

Only 85GB uncompressed for current revisions, excluding “user” pages. Not so bad!

Reply on news.ycombinator.com