Hacker News Reader

Comment by seestem | Hacker News Reader

It would be better if the Internet Archive was decentralized without a central point of failure, maybe run on something like bittorent.

View on HN

You say this as if it is an original idea. Of course the IA is working on this and have been for over 6 years. There already is a DWeb version. They have been advancing DWeb infrastructure. The IA hosts all kinds of DWeb developer events.

But it is over 50 petabytes and the IA gets a huge amount of traffic through the regular web that they need to serve quickly and efficiently to their users.

Guess what has happened over 6 years of decentralization of 50 TB? People only seed what they want or care about and there aren't enough seeders to host. They set all this up and nobody volunteers. You're a DWeb advocate and you haven't been seeding. That's a recipe for disaster if they rely on the goodness of volunteer seeders. The IA's mission is broader. DWeb will ever only compliment the IAs mission.

https://blog.archive.org/2021/02/18/behind-the-scenes-of-the...

https://www.bleepingcomputer.com/news/technology/archiveorg-...

How does one contribute? In the article you linked:

> there is no information on how users can get involved in the decentralized version of Archive.org and who the peers are that are distributing the content.

The other link doesn't mention how people could help host data either. If there is a way, then it seems like more of a marketing issue if those willing are unaware or unable to figure out how. I can't find any actionable steps on how to contribute.

edit - it seems the dweb version was a frontend for archive.org testing serving IA content over alternative protocols. It was never finished or expanded on unfortunately. Links to it are dead but here's the github repo https://github.com/internetarchive/dweb-archive

Can confirm that issue about people only seeding what they are interested in.

I found a dataset I wanted to hoard but the authors website was gone. A dataset site had a torrent and I said great I'll just torrent and seed that and help keep the thing alive, turns out I can't find a single seeder for the torrent.

If I help seed this DWeb and it turns out it has some copyrighted materials in it, will I be potentially held liable?

This.

Until a clear, precise answer to this question is available, it is unreasonable to expect individuals to take risks and seed.

It is one thing if an organization like IA gets in trouble with the law. They have money, lawyers, name recognition and are big enough to at least fight a lawsuit, even if they lose. Who is going to help an individual if he/she gets in trouble with the law, unknowingly? Am I expected to read through tons of complex copyright law and interpret it, just so I can seed a handful of items? No thanks.

You're always responsible for what you, yourself and your computer does. There is a chance EFF/some other organization could help you out in case you end up in court, but that's a maybe, not a guarantee.

Harder to make this argument with encrypted distributed filesystems. If I'm storing a single chunk of an encrypted blob on Filecoin, am I responsible for the entire file even if I don't know what's in it, and I'm only storing a single fragment?

This depends on the jurisdiction you're in. I.e. Europe's GDPR argues that you need consent to keep someone's personal data. Encryption doesn't equate anonymization, so there's a potential liabity.

It seems to me the various efforts are dead or stalled. Anything in actual current development or production? IPFS was supposed to go in that direction and still exists, sure, but not to provide IA duplication (that is advertised.)

https://www.reddit.com/r/DataHoarder/comments/h02jl4/lets_sa...

I’ve always been fascinated by this post.

You're describing a network effects problem, specifically a collaborative game failure. Need some mechanism designers and big tech cos to jump in, stat!

It is not my original idea, but it is an obvious idea to anyone who knows how the internet works, I just added it to get the discussion going.

It already exists. IA has had a DWeb for 6 years. Nobody seeds it.

If nobody seeds it (or continue development) then it's dead. Inspiration and perhaps code for the next effort sure. But not "exist" that makes a difference.

Incentivising seeding is hard. Maybe cryptocurrencies can be useful here, but I understand not everyone likes them especially here on HN. In retrospect the ideal setup would have been if archiving was included into the core HTTP protocol.

Cryptocurrencies implies that people would pay / get paid for it... just pay the directly IA then, or your own servers. Cryptocurrencies imply someone's skimming off a lot for their own pockets.

I meant more like in bitcoin how the miners get paid for mining, or how validators are rewarded in proof of stake blockchains. This techniques can be used to incentivised seeders.

Maybe someone can invent a proof of seeding protocol? So that would bring some good to the public instead of just burning energy. Don't ask me how it would work...

Storj, Filecoin, etc. fill this gap but it's still really hard to earn enough to justify the effort at small scales.

i'm doubtful that whatever crypto incentive that is offered will make up the cost for me to DIY this in my home. which is why crypto miners scale, making crypto a centralized system. i don't care what the latest white paper says, it is still controlled by few people and not decentralized. in the same way that the ussr replicated the centralization of american capitalism, crypto replicates the centralization of trust while marketed as something else.

The problem is that it's hard to do this in a way that ensures good archival of ALL resources.

Bittorrent works well for popular things but fails for marginal content (unless some really dedicated individuals step in.)

What the internet archive provides is a way to have access to many many resources which you didn't know you needed in advance.

Lol, in a way the decentralized version is actually the internet.

Or a brain.

I'm working on this, ArchiveBox v0.8 adds the beginnings of a content addressable store, with plans for bittorrent-backed instance-to-instance sharing in a later version.

I think Archive.org should still exist too (and ArchiveBox donates + submits URLs to Archive.org too), but having a self-hosted option where you can archive personal stuff that requires a login, and do P2P sharing with with fine grained permissions is a gap that should be filled.

Aiming to archive the entire internet is Archive.org's goal, aiming to archive the part of the internet YOU care about is our goal.

I'm hoping that Autonomi (formerly The Safe Network) is up for the job when (if) it makes it out into the real world one day https://forum.autonomi.community/t/the-internet-archive-a-pe...

[I know that some percentage between 95 and 100 of crypto projects are a scam. I personally believe this one isn't, after much diligent reading. Whether it gets released or does what it claims it will do is another question, but please do spare me the kneejerk anti-crypto reactions, if you can. Just because they're almost all money-making scams, doesn't mean they're all money-making scams.]

That would be an awful lot of replication or very shitty archive. Decentralization works when each node can serve all the functions and content alone or when you don't care about completeness.

Unless I'm missing something, an archive is not something small or something that's just as good when part of it is missing.

I kind of agree, but the way the internet is going, with everyone being behind carrier-grade nat, it's not much of a decentralized network of computers anymore, not to mention all the kids with their laptops and tablets not even hosting anything :(

There are ways around this, I've experimented with setting up a cluster of ArchiveBox instances that share snapshots over Tailscale. Tailscale lets users sign up for free accounts, and you can share machines between separate accounts. A (CGNAT-compatible) decentralized invite-only network could concievably spread that way.

UPnP exists and allow devices to ask the router to open a port to them.

UPnP is useless with CGNAT (Carrier Grade NAT), which is what the op is talking about.

There are other ways to get seeding working, though, including IPV6, which is gaining adoption, so I don't agree with the OP.

UPnP is just automating the process of forwarding ports, CGNAT will still screw you sideways because you're behind a router you can't access or order around.

That doesnt help with CGNAT.

that will never happen. no one is going to be able to seed the amount of data that IA has. The only thing they can hope for is that a company like Google or CF provides another data center for them.

I don't know if bittorrent has improved - but 20 years ago I had a personal issue with it.

At that time our son was using it for games. He goes away to college and came home for the first school break. I get a phone call from our internet provider asking if our son was home. I was so shocked and handed the phone to our son.

Apparently at that time bittorrent was optimizing for the most efficient path to a host. Since we had relatively good connection, the mighty weight of the internet was funnelling through our tiny internet provider to our son's computer. The provider (without our knowing it) had made a deal with our son that he would only turn on bittorrent between midnight and 6 AM. I doubt other providers would be so generous.

I have been sceptical of bittorrent since that day.

All clients today (and probably back then) have options to limit bandwidth consumption including throttling, scheduling, and total data transfer caps. For serving mostly HTML and images, dedicating even 10% of a home broadband connection to serving content would allow many, many people per day to access archived pages.

Reply on news.ycombinator.com