Hacker News Reader

Comment by petra | Hacker News Reader

Actually torrents are pretty bad for storing "rare" files. Maybe someone should offer a fix.

I think this is mostly a consequence of how the clients work. When I download something with BitTorrent, it goes into a Downloads folder. Then I want to move and rename the files to fit into an organized collection, but then the client won't be able to find it anymore. If my client were able to track files across moves and renames, I'd be much more willing to seed things permanently.

(It would also need to have good upstream-bandwidth management, use a number of TCP connections that's less than O(n) in the number of torrents, and offer a few bulk operations like "stop seeding torrents with more than N seeds" and "resume seeding torrents with fewer than N seeds".)

I've "solved" this by having torrents go to a "Torrents" folder, then symlinking those files into my organized collection.

I do the same thing but into multiple folders with a unionfs in read-only mode so I don't accidentally modify the files and cause them to be re-downloaded

I hardlink them instead. Then I can prune them from the torrents folder without affecting the organized file.

Even if I stop seeding, I like to keep them in the torrents folder because it gives me essentially free integrity checking.

You can deal with the renaming issue with a bunch of symlinks. It's pretty easy to script that up.

To make it work i think the client should have some hard drive space allocated for the purpose of "preservation" , which he can automatically use according to some reasonable policy(maybe only files i downloaded can be put in the folder) and it should allocate some share of my bandwidth for that purpose. Than the client automatically manages everything with coordination from the network.

That's the only way this can scale for regular users, i think. And of course it may use social mechanisms like feedback "you're the reason why this rare work of art if available for everyone".

As for scalability of TCP connections , you're probably right, not my area of expertise , but once this will be declared as an important BT problem with great practical value, i'm sure some researchers would be happy to contribute and might solve this.

There are distributed data stores that work just like this. IPFS (https://ipfs.io/) is a recent effort, while Freenet (https://freenetproject.org/) has been around for a while.

Whether this can be retrofitted into Bittorrent not clear. Conceivably one could write a bridge between IPFS and Bittorrent such that an IPFS url could serve as an "IPFS seed" much like a HTTP url can serve as a BEP 19 webseed.

It could work, but wouldn't it be simpler , as a first step, to add this feature in a basic form to a python based BT client like tribler and see where it goes from there ? I.e.

1. At torrent delete , set a flag on torrent to hide

2. At torrent display , don't display hidden torrents

3. Periodically or on storage_full event - fully delete hidden torrents , based on some simple strategy like "remove those with most seeds" ?

And after building it and testing a bit , creating some nice web page , a bit of marketing , and gather developers/users ? Who knows, might become popular .

Might have tried to do so myself , but i'm not really a developer.

Anyway, is it realistic?

You'd want to write a server that could retrieve and serve the file over bittorrent, but would store it canonically in IPFS.

In the good old days of using (Napster|Audiogalaxy|Gnutella) I could point the file-sharing application to my collection and presto - it was all shared and available to whoever was interested. Nowadays, I just share whatever torrent I have participated in lately, plus a few that seem popular but rare (those most uploaded from my client) - but I have a whole lot of rather rare music that I fail to share for lack of a way to do it. Is there a contemporary way to share a whole collection or is that really missing from the toolset ?

Well - maybe I'll finally get around to setting up a heeadless gtk-gnutella node again.

eMule and derivated have the .emulecollection format, which is just a binary file grouping together multiple ed2k links to files.

As for the network eMule/KAD seems do be doing ok with 200K+ peers while Gnutella nowadays seems to be populated mostly by slightly-compatible chinese clients, which bootstrap from traditional gnutella but are incompatible.

Soulseek is still up and kicking.

I thought about it and found it a bit too social and too insular - having "rooms" as a central feature feels odd to me who would rather use a global search of the whole file-sharing network.

Also, having to run a desktop client was putting me off. But then I just found http://www.museek-plus.org so I might try that... But it is disappearing from Debian: https://packages.qa.debian.org/m/museek+.html - not a good sign...

But there is a global search of the whole network, chat rooms are separate thing.

Well, it does not match wcd at all anyway.

The problem is not really torrents. The problem is that seeds are ephemeral. But an equally ephemeral source for any protocol will always make the availability of the resource low. The solution is not to change the protocol; the solution is to have less ephemeral sources.

It would be interesting to see how a Usenet/Torrent adapter may work to at least supplement loss

The Internet Archive might be a good host for rare (and not copyright infringing) torrents.

It actually is, because it already provide torrent access to its content. See, for example, the Matrix ASCII (only the sample though):

https://archive.org/details/TheMatrix-InAscii

(1 more point for how terrific the Internet Archive is)

EDIT: it turns out the complete Matrix ASCII is hosted on the IA, making its availability a non-problem for years to come, even though it's not the original torrent anymore

Reply on news.ycombinator.com