Hacker News Reader

Comment by Mizza | Hacker News Reader

This is an okay idea, but it doesn't really solve the big problem facing BT right now: content discovery.

Mutable Items are good for updating content, but that doesn't matter if nobody can find it.

There needs to be a BEP for a way to host, serve and search metadata about torrents, not just their info-hashes. This should be priority #1 for BT devs. Fortunately, there is at least some related thinking in this direction: http://www.bittorrent.org/beps/bep_0044.html although I don't think that it really goes far enough, as there is no standardization around how the data for searching will be structured, etc.

View on HN

Tribler is a research project of Delft University of Technology. (...) Tribler is the first client which continuously improves upon the aging BitTorrent protocol from 2001 and addresses its flaws. We expanded it with, amongst others, streaming from magnet links, keyword search for content, channels and reputation-management. All these features are implemented in a completely distributed manner, not relying on any centralized component. Still, Tribler manages to remain fully backwards compatible with BitTorrent.

https://www.tribler.org/about.html

I worked on an anonymous P2P research project many many years ago now, and we were shocked when we saw how much funding they received compared to us (3 mil EUR!) - but I'm super happy to see that their project is still alive and kicking! I guess the money was put to good use.

Torrent sites could publish their database dumps via this extension, and consumers would automatically download their updated index. A torrent can be anything; it can also be a list of other torrents ;) So it does help with content discovery.

An index? That could be huge. I think it's senseless.

So many p2p protocols have support for p2p search (to my mind comes Kademlia), why can't BitTorrent have p2p search?

Vuze (formerly known as Azureus) has (or at least had a prototype of) p2p search.

I wrote a paper on building a torrent search index quickly using the DHT records that this feature emitted at the time. [link redacted]

Great work with this. Are you still in academia doing research?

The Mainline DHT (the more-supported DHT vs. Azureus/Vuze DHT) traditionally only contains (iirc, hashes of) torrent infohases mapped to peer lists. With BEP 44 (arbitrary data in DHT) or some other enhancement, this may change, but right now even if you traverse the DHT there's no "interesting" metadata about the keys (or, y'know, the actual content of the .torrent file) inside the DHT.

From the sibling comment (posted by swolchok)'s paper, a relevant quote:

"We chose to demonstrate our proof of concept only on Vuze due to the significant additional complexity of supporting two DHTs in the crawler. While Mainline does not contain torrent descriptions, its peer lists are keyed by the torrent infohashes themselves, so no additional machinery would be needed to discover infohashes. However, some method would be needed to discover torrent names, such as downloading .torrent files directly from peers using the metadata exchange protocol.

To explore this possibility, we built a prototype .torrent crawler. Since we are operating in the context of the Vuze DHT, our DHT crawler needs to obtain both a torrent description and a peer list before the .torrent crawler can contact the appropriate peers with the correct infohash. However, the .torrent files contain useful additional data, including the full listing of files contained in the torrent"

Kademlia search is also rather useless as it is flooded with false results. This is of interest not only to malware writers but also as a service to sell to rights holders.

You don't see a problem with that..?

You still need to have the entry point - that's exactly where the censorship is occurring. Try to find a copy of the KickAssTorrents database on Google right now and tell me how easy it is.

The approach you're talking about is essentially what http://bitcannon.io/ is trying to do - but that requires running a separate app, a local MongoDB server, downloading many gigabytes of database data, and half an hour to import the torrent DB into your local instance, and that STILL requires you to be able to acquire the Torrent database or related infohash, which isn't easy because of the very censorship we're trying to sidestep.

Quite frankly, that's a really dumb solution to this censorship and it's naive to say that it's useful. This will never be used in the fashion you're describing and it does nothing for anti-censorship.

It makes so much more sense just to have the DHT be actually useful in surfacing content.

> You still need to have the entry point

Which will now be a short hexadecimal code that is easily shared and can't be taken down.

If KickAss decides to share their dump using their public key, it becomes much harder to censor because of how the DHT system works. Authorities would need to shut down all the nodes that republish that public key on the DHT, which is incredibly hard to do. Much harder than asking ISPs to shutdown a domain name in DNS.

I remember someone setting up a bittorrent search engine based on DHT scraping. It's probably been taken down, but I think it was called DHT Dig, and there's apparently a library with that name.

Something like that would make it easy to create centralised search engines for casual users, and allow power users to run local databases.

There was btdigg at some point (https://btdigg.org/) but for some reason it's down today. The approach is not particularly complicated, there even were papers about that:

* In Vuze: https://www.usenix.org/legacy/event/woot10/tech/full_papers/..., slides at https://www.defcon.org/images/defcon-18/dc-18-presentations/...

* In Mainline: https://arxiv.org/ftp/arxiv/papers/1009/1009.3681.pdf

Basically all you have to do is sit there, people will come to you asking for content, you'll ask as a standard DHT node but note that the infohash exists, and get more information (name, number of peers, ...) about it in the background. All you need is a few IPs.

Ah yes, BTDigg was the name.

I'd imagine to do more thorough scraping you'd want to reconnect to the network repeatedly with different peer hashes, otherwise you'd only see the infohashes "near" one point in the network.

Yep, that's the idea: if you join the DHT network with multpile, far away ids, you get more requests coming in. Unfortunately there is BEP 42 (http://www.bittorrent.org/beps/bep_0042.html) which restricts the number of IDs you can have on the same IP (the restriction is there for a good reason, so it's a good think to follow it).

Reply on news.ycombinator.com