Hacker News Reader

The "v2" torrent file format allows most of this. Some clients have support already.

All you'd have to do is make a torrent of your whole hard drive and then seed that. You don't need to publish the torrent anywhere.

If anyone else in the world is downloading any other torrent that happens to contain a file you have, they will end up connecting to your machine to download it.

A v2 torrent allows clients to identify duplicate files across torrents via the "pieces root" entry in the metadata, so if they're downloading from torrent A and B, and each share file C, they can utilize peers from either swarm.

But there's no way for other clients to know that there exists another torrent containing the file they are interested in if they only have the metadata for torrent A. In other words, there's no lookup mechanism for a "pieces root" to know that torrent B exists and contains file C.

If you were to make a v2 torrent of your entire drive, other clients won't know to download from your torrent. They'd need to have the contents of the metadata to know it would contain a file they are interested in, and have no way of knowing which metadata contains the desired "pieces root" entries without downloading all of them.

I'm very interested in this problem space, if you are aware of clients/mechanisms that allow for this I would love to hear them.

I don't think you can solve this problem, without destroying the system by releasing too much data.

If I find a torrent of a file, and can look up all the seeders across all torrents for this file... then so can the FBI.

> there's no lookup mechanism for a "pieces root" t

I guess they'd just have to publish each pieces root in the DHT. And obviously you would let the user decide which directories to share...

I guess that'll have to wait for torrent v3...

This.

Also, advertising every file in torrent (of unsuspecting user) to entire world would be a HUGE privacy flaw.

But you already advertise (via the DHT, which offers no secrecy) the torrent file... and that contains all the data needed to get all the files within the torrent.

So this data is already being published, just in a less queryable form.

Hm, is there a security concern here?

Someone could use this to

A) Remotely check the presence of any specific file on your machine.

B) Exfiltrate the contents of any file they know the hash of (or possibly more specifically, the hash of each piece? I don't know the protocol details).

Fine if you have a dedicated "I expect the contents to be public" drive or directory, but not something I'd want to do on my OS drive.

I have seen people obviously sharing their entire hard drive on P2P platforms, and it's usually been someone's C drive and a massive security issue. I don't think some clients do enough to make it clear to people that they need to be careful not to share more than they intend to or should.

On modern windows, ideally there are no sensitive files accessible to a user program, because everything should be in the secret store/protectedstorage.

Obviously a lot of programs still store cached session keys and passwords in plaintext somewhere in your user directory though...

There's a lot more to be concerned about than session keys and passwords. Tax documents, health records, photos, private journal documents, just to name a few.

besides that a user's Photos and Documents directories can be rather sensitive

and assuming nobody puts their passwords.txt on desktop and keeps unprotected wallet.dat around.

I did work on a proof of concept program to accomplish this for my own content library. It would scan a directory to find files and compare them with locally stored metadata. For v2 torrents this is trivial to do via a "pieces root" lookup, for v1 torrents it involves basically checking that each piece matches, and since pieces may not align with the file then it's not possible to guarantee that it's the same file without having all of the other files in the torrent.

I built it with libtorrent and after loading in all of the torrents (multiple TBs of data), it would promptly and routinely crashed. I couldn't find the cause of the error, it doesn't seem it was designed to run with thousands of torrents.

One problem that I've yet to build a solution for is finding the metadata to use for the lookup phase. I haven't been able to find a publicly available database of torrent metadata. If you have an info hash then itorrents.org will give you the metadata, if it exists. I started scraping metadata via DHT announcements, but it's not exactly fast, and each client would have to do this unless they can share the database of metadata between them (I have an idea on how to accomplish this via BEP 46).

I have a solution to this, it's the successor to Magnetico.

Could you please share a link to your solution? I would be interested to take a look

>One problem that I've yet to build a solution for is finding the metadata to use for the lookup phase.

I think BEP 51 followed by BEP 9 is all you need.

This is how I was originally achieving this. As I said, it's very slow. I don't think it would be a good solution on its own because it would require that every client be constantly sampling all DHT nodes, and downloading all metadata and indexing it for a potential future lookup. It's a huge amount of additional load on the DHT.

I think a better solution would be some way for clients to query the DHT for a specific "pieces root", but I don't know if all clients publishing "pieces root" for the torrents they know about would also be a good idea. Some kind of distributed metadata database where clients can query would be ideal.

Would you mind sharing the source? Sounds like something others could build on.

The source is available here: https://github.com/chhs1/content-seeder

But why? The current Kademlia implementation built in eMule/aMule does exactly this. Primary data advertised is file name string, there is also a support for several meta fields. Transfer proto sucks, but I don't think it really matters in the case of ultra-rare content.

That `ultra-rare content`, eMule is mess if you need collection of files, ofc you can share ZIP but they get somehow repackaged too often. Torrents nicely group the stuff - it is kind of album.

I've dreamed of federated meta-client which mixes all available p2p networks in a wild and can download missing torrent part from eMule or Soulseek.

> I miss a p2p application with torrent packaging and Kademlia like per-file advertising and discovery, where I could point it to my hefty NAS directory of random things and they could be wired to released torrents.

That was the problem with p2p no one bothered to actually package their releases instead just slinging it onto the internet no care given as to if it was complete or even correctly labeled.

Sites like pirate bay had everything, torrents are a super fast and error resistant way to distribute content its the perfect system. The problem was that the content is illegal so it was nuked by companies with more money and power than any one can fight against.

The problem has never been technical or distribution the issue is the law.