Bitmagnet: A self-hosted BitTorrent indexer, DHT crawler, and torrent search
bitmagnet.io
bitmagnet.io
Heh. That was one of my first projects when I was still learning to code back in 2012: https://github.com/laino/shiny-adventure
The DHT crawler/worker lived seperately, and I eventually put it here to rescue it from a dying HDD: https://github.com/laino/DHT-Torrent-database-Worker
The code is abhorrent and you absolutely shouldn't use it, but it worked. At least the crawler did - the frontend was never completed.
Since the first implementation of mainline DHT appeared in 2005 and crawling that network is really quite an obvious idea, I doubt we (a friend was working on it as well) were first either.
Today was the first commit after 11 (9 oct 2012) and 5 years (24 nov 2018), respectively, on the projects. I think your repo might be part of some sort of oldest 'active'- or 'not ported to another repo'-repo
For what I've found in ~10 min (google/gpt), excluding git projects existing before spring 2008 (couldn't get a quick consensus on feb vs april of that year), there's not a lot
(I'll edit this part if sources are requested)
It holds several one file python script experiments and toys that I lumped into one place to get them off my hdd and make them available from wherever. Recently remembered it existed and added another one. And while I was in there I also ran 2to3 on the ones that needed it and polished the results up.
The issue and my response on GH: https://github.com/bitmagnet-io/bitmagnet/issues/11
I have started the docker-compose.yml file in WSL and it has been running for an hour slowly accumulating a few megabytes of redis data running at about 5% CPU usage. Inspecting it shows magnet links. It appears to be working.
But visiting the the web interface at localhost:3333 just yields "Firefox can’t establish a connection to the server at localhost:3333." after a 30 second timeout.
Would you have a guess why?
Bad actors - whoever they may be - need to see your rights constantly reasserted.
If you want to help pay for lawyers to fight those DMCA notices with counterclaims and lawsuits, put up or shut up; the FSF, EFF and ACLU have been noticeably disinterested in doing so.
It's about arguments in court about the intention of the software if sued. Images of copyrighted content indicate intent to infringe copyright. Without those, you can argue it's only meant to find and index Linux image torrents or whatever.
Fair use doesn't enter the picture at all.
"It then further enriches this metadata by attempting to classify it and associate it with known pieces of content, such as movies and TV shows. It then allows you to search everything it has indexed."
[1] https://github.com/sergiotapia/magnetissimo
> The DHT implementation was largely borrowed from Magnetico (https://github.com/boramalper/magnetico) a popular and widely used app which is likewise unable to service these requests.
Torrentinim is the successor of Magnetissimo but it lacks Radarr/Sonarr integration (there is a pull request for Torznab support for both). Spotweb has Newznab support [1] but at Black Friday (soon) there's usually tons of deals available for Newznab indexes.
I don't care about BEP51 as I don't have huge upload. That is also why I prefer Usenet over torrents. But torrents are a useful and sometimes required backup mode. Just not my preferred one.
[1] https://github.com/Spotweb/Spotweb/wiki/Spotweb-als-Newznab-...
https://torrent.fedoraproject.org/
I just want a simple list, and the backend.
I use this service to do security research a fair bit. It'd be nice if there was a higher quality self-hosted version I could use so I'll be watching this project with interest!
But I am curious what you mean when you say you use it "to do security research"?
Are you just looking for security information that is available in torrents, or does btdig have some other features that I am unaware of?
IIUC peers favor nodes they've had longer relationships with to provide stable routes through the DHT.
This means short-lived nodes receive very little traffic, nobody is routing much traffic through fresh nodes, they choose nodes they’ve had longer relationships with.
The longer you stay up, the more you start seeing.
At least this is what I've observed in my projects. The only way I've been able to get anything interesting out of the DHT in the last ~5 years has been to put up a node and leave it up for a long time. If I spin up something, the first day I usually only find a handful of resolvable hashes.
Not to mention it seems the BitTorrent DHT is very lax in what it will route compared to other DHTs (like IPFS) meaning many of the hashes you receive aren't for torrents at all.
It's using roughly 6% CPU time for the crawler and another 1-2% for postgres, on a second-gen i7.
As a datapoint to set expectations: 4000 torrents have been captured so far, and somewhat surprisingly, they're not very current results, necessarily.
For example, a certain wildly popular TV series about samurai in space swinging very hot swords around which just had its season ending episode last night (I think)...that ep isn't in my list so far, but the episode prior to it, and the first two episodes, are.
There's a ton of random, low-seed torrents, so it's actually kind of interesting to search by type, year, etc and see what comes up.
Edit: found the init schema and things seem to be working now: https://github.com/bitmagnet-io/bitmagnet/blob/main/migratio...
It would be really nice to be able to sort by header (size, seeders) and/or some filters for seed/downloaders (for example, filtering out anything with less than X seeds.)
I've thought about this problem a lot. Having a federated / distributed tracker but with some form of trust based, or opt-in curation would be amazing.
The "Trust framework" that everybody is after: lightness of tit-for-tat with little overhead. With trust that actually can make The Internet a nice place again! Curation is indeed the problem discovered by HN practitioners such yourself. However, nobody in academia is working on this. As one old scientific article is titled: "Curation, Curation, Curation"
It's now archived due to effort being redirected to Bitmagnet.
No, it ignores all incoming requests. You don't need special software to help out, just run any normal Bittorrent client in the background (no need to download or share anything) and it will help out. Just make sure you forward the right port if you are behind a NAT. Traffic will slowly increase over time, and drop quickly when offline, so leaving it up for days at a time is better when possible.
(In a sense, I suppose this would still be a "daemon", but that daemon would be the map-reduce infrastructure.)
All other nodes are operated by someone else, so they don't cooperate on anything beyond what the protocol specifies. Which means everyone is their own little silo. If you want a list of all currently active torrents (millions) then you have to do it with 1 or a handful of nodes, depending on how many IPs you have. DHTs are not arbitrary-distributed-compute frameworks, they're a quite restrictive get/put service.
BEP51[0] does let you query other nodes for a sample of their keys (infohashes) but what they can offer is limited by their vantage point of the network so you need to go around and ask all those millions of nodes. And since it's all random you can't really "search" for anything, you can only sample. And that just gives you 20-byte keys. Afterwards you need to do a lot of additional work to turn those into human-readable metadata.
> DHT indexing already is possible and done in practice by passively observing get_peers queries. But that approach is inefficient, favoring indexers with lots of unique IP addresses at their disposal. It also incentivizes bad behavior such as spoofing node IDs and attempting to pollute other nodes' routing tables.
If you have a lot of IP addresses (from e.g. AWS Lambda) then you can partition DHT keyspace across a large-N number of nodes and then very quickly discover everything in the keyspace.
The trick is that, since BEP51 exists, you don't need to have all these nodes register themselves into the hash-ring (at arbitrary spoofed positions) to listen. You can just have all these nodes independently probing the hash-ring "from the outside" — just making short-lived connections to registered nodes (without first registering themselves); handshaking that connection as a spoofed node ID; and then firing off one `sample_infohashes` request, getting a response, and disconnectting. The lack of registration shouldn't make any difference, as long as they don't want anyone to try connecting to them.
Which is why I say that these are just "crawler agents", not "nodes" per se. They don't start up P2P at all — to them, this is a one-shot client/server RPC conversation, like a regular web crawler making HTTP requests!
But neither of these approaches is what I'd call a "super simple DHT crawler CLI tool" that the initial comment was asking about. BEP51 is intended to make crawling simple enough that it can run on a single home internet connection, but a proper implementation still isn't trivial.
How do you guys handle that risk?
You wrote "works" twice
Also what happens if illegal content gets scooped up into the index?
The question is whether Law Enforcement and "Intellectual Property" watchdogs make a meaningful distinction between the two in their monitoring tools.
so isn't the answer then "they'll continue to police it the way they already do"? i don't know what a MaidSAFE is, but the context of this discussion is the DHT, and so public (indexable) torrents, and so however you encrypt the content doesn't matter because you have to provide the decryption method to anyone who asks for any of the previous context (public torrents/indexes) to make any sense.
The weak point seems to be the tracker or filename, but been told https hides that so not sure.
If you listen to BitTorrent's DHT network, you'll build an index of everything shared on BitTorrent (over time), this will include commercial movies and such.
Hi, I worked on gnutella and lots of P2P systems in the early 00s. This will devolve into noise and spam as the number of users who adopt this feature pass a critical mass. With a fully decentralized system, there are no gatekeepers, and as such, there is no way to filter counterfiet items. While your client will present with you the data you are searching for, you will find out (usually hours later) that your supposed pirated download is actually just a 2hour loop of Rick Astley (still piracy though, so you are still winning.. i think?).
The spam problem isn't nonexistent within the centralized services either.
What i'm saying is it becomes a numbers game and those fake seeders usually have deep pockets financed by the content creators themselves
https://news.ycombinator.com/item?id=37779341
> New releases of something that just came out tend to suffer from this, though. Sometimes the counterfeits reach escape velocity - the rate of people joining in downloading the counterfeit exceed the rate of people realizing and stopping, thus giving the illusion of a legit torrent.
New releases of something that just came out tend to suffer from this, though. Sometimes the counterfeits reach escape velocity - the rate of people joining in downloading the counterfeit exceed the rate of people realizing and stopping, thus giving the illusion of a legit torrent.
Currently this problem is being solved by torrent sites' reputation and comment systems. If we imagine a world where only decentralized indexes like Bitmagnet exist, your prediction is 100% accurate. This only works if reputation from a reliable site is bootstrapping the initial popularity of a torrent.
You are describing a pay-to-play model. The validator is if the seeder/leech count is high. Well does DHT provide aggregate bandwidth of each torrent? If not, you can easily spin up 1000+ nodes and connect to your torrent. Tada fake popularity. If bandwidth is known, then you simply raise your costs a bit by running fake clients. There are anti-piracy groups who's entire mandate is to provide noise in the piracy ecosystem. Food for thought: bandwidth costs for this would be a rounding error for e.g. MGM, Universal, or any major content creator.
DHT does not offer any sort of reputation or comment system. Back to centralized torrenting which is why I suspect DHT crawling has not been a very popular feature
Sure, but like the other commenter said, this has been possible for years, and yet public trackers aren't swamped with fake torrents. I think in all my years of using BitTorrent I've only ever found a single fake torrent, where the content was inside an encrypted RAR with no key (obviously there was no way to know it was encrypted ahead of time).
If the criticism is that a DHT crawler is going to be more subject to this than a website where people submit upload torrents, that may be the case, but I think the author of this project underestimates the DHT crawling going on. I believe the torrent ecosystem is largely automated and there's little in the way of manual submission or human review going on.
That is, the usefulness of DHT crawling is inversely proportional to how many people are doing it.
What you need is sorting and categorization. If you really want to involve authoritative opinions on metadata, then use a web of trust.
I ask, because a similar heuristic is used in crypto wallet software, visibility-weighting the various "versions" of a crypto token with the same metadata, by (in part) which were oldest-created. (The logic being: scam clones of a thing need to first observe the real thing, before they can clone it. So the real thing will always come first.)
Of course, I'm assuming here that you're searching for an "expected to exist" release of a thing by a specific distributor, where the distributor has a known-to-you structured naming scheme to the files in their releases, and so you'll only be trying to rank "versions" of the torrent that all have identical names under this naming scheme, save for e.g. the [hash] part of the file name being different to match the content. This won't help if you're trying to find e.g. "X song by Y artist, by any distributor."
Correct me if I'm wrong but as far as I understand, passively listening on DHT would only mean you build up a list of infohashes of everything shared on BitTorrent. You'd actually have to reach out to your DHT peers to know what files the infohashes actually represents.
Wrapping back to grandparent's question of
>Also what happens if illegal content gets scooped up into the index?
I think this could get dicey if someone announces something very illegal like CP, and your crawler starts asking every peer that announced the infohash about it's contents with this[0] protocol. This would put your IP into a pretty awful exclusive club of
A, other crawlers
B, actual people wanting downloading said CP
Yes, you're correct! I should have stated that, you still need to resolve the metadata from the peers that have the infohashed files hosted. That's a separate operation from downloading the file's content.
Of course, respecting these options in the torrent file is still up to the client. This is one of the reasons why all private trackers have a client whitelist too.
> In-place seeding: identify files on your computer that are part of an indexed torrent, and allow them to be seeded in place after having moved, renamed or deleted parts of the torrent
Does anything do this already? It would be amazing to point a client at a folder of unstructured junk and have it magically find the right parts.