Hacker News Reader

Comment by CodesInChaos | Hacker News Reader

In my experience public torrents often die as they grow older. It doesn't help that BitTorrent V1 makes long term seeding annoying, and BitTorrent V2 is almost never used.

View on HN

I never understood this, is there anything that makes it difficult for the original uploader, the one that supposedly offers the file directly, to offer a torrent instead for the same amount of time?

As far as perennity is concerned it seems strictly better.

If they're no longer using that model they may not be willing to continue using their storage for it.

How does this address the previous point? If they are not providing storage, then centralized or decentralized doesn’t make a difference.

Torrent/P2P can only add redundancy, so it’s impossible to have worse availability than a download link?

yes, but a few dedicated hoarders could keep many models alive, and I'm fairly certain the local LLM community has plenty of people who would.

They don't. That's the conversation topic.

Nor do they need to. 99% of everything is crap, and not worth prescribing except for a random sample so future historians can study our crap.

Every change to the source is effectively a new torrent. This creates a ton of fragmentation as data is reorganized, remixed, reencoded, and so on.

You can see this with many Linux distros: there is no single Debian torrent that people seed for years because there's always a refreshed version.

Distros are a bad use case for P2P anyway since you depend on upstream as soon as you start upgrading and installing packages.

> Distros are a bad use case for P2P anyway since you depend on upstream as soon as you start upgrading and installing packages.

This is true for any distribution method not just p2p. You can even download a nightly through torrents so what does it matter how the data is transferred if it’s always going to require `apt update`?

Yeah, I just use the "netinstaller" ISOs since it's much smaller and never needs to be updated. If I had a need for air-gapped/offline installs I'd either download a larger ISO or just manually install packages from .deb as needed.

You can trivially have storage deduplication for the files served via torrent, transparent to the protocol. The most trivial version of this that you can do today with pretty much any client is having a single directory containing files serving multiple overlapping torrents.

> files serving multiple overlapping torrents

This sounds wildly complex, especially from a discovery perspective.

I don't see why it would be. It's transparent to other clients just like it is to the protocol. It cannot be more complex than alternatives by construction.

Or use a filesystem with dedupe like ZFS.

but model releases are already non-changeable?

Yes, models are a good use for P2P especially if everyone agrees to share the same torrent and someone (or a cohort) commit to seeding for the long haul.

IPFS has a solution [0] to this problem

[0] https://specs.ipfs.tech/ipns/ipns-record/

Most of IPFS doesn't actually work very well, if you've ever tried to use it

> if you've ever tried to use it

Heh, you got me :) IPFS is one of those things that I love reading and about and thinking about using someday, but somehow never get around to it.

The problems begin with taking 5-10 minutes to locate a file on the network. That's right, when you ask for a file it takes 5-10 minutes. Also if the file isn't in the network at all then it never terminates.

Nobody noticed because everyone just used the central web gateway that cached every file anyone ever accessed.

You can add an existing HTTP download as a "web seed" to a torrent, so they don't actually need to do anything for people to share it as a torrent.

The biggest problem with BTv1 was the lack of per-file checksumming, and swarm merging (i.e. individual files have shared seeding pools across torrents). BTv2 specs the latter, but I think only BiglyBT actually implements it. Having both of those features from the get-go would've gone a LONG way to fixing the dead torrent problem.

A torrent with a webseed is strictly more resilient than a direct download link alone.

You only need one person/organization to commit to seeding. The majority of people do not want to seed at all without some sort of incentive.

If this site represents a coordinated datahoarding effort then there will be at least a few people who will seed indefinitely.

The last guy (kimdotcom) who was working on this (incentive for seeding) seems to be heading to the US: https://www.rnz.co.nz/news/science-and-technology/651123/cou...

It’s interesting he’s no longer getting any media attention any more.

Yeah but that's what provider like huggingface could just do, keep seeding the models so they are still accessible

I always wondered why v2 is never used... you can even search for files by their individual hash with it.

Because history is path-dependent, as engineers keep learning over and over again. It doesn't matter whether Plan9 is theoretically superior to Linux - we're all on Linux and nobody's porting all the apps over.

Reply on news.ycombinator.com