OpenBitTorrent – An Open Tracker Project
openbittorrent.com
openbittorrent.com
https://github.com/ngosang/trackerslist
trackers_best is usually sufficient.
I've written a Chrome extension which periodically grabs the list of active trackers from NewTrackon and automatically adds them to magnet links as you're browsing. It's made torrenting so much faster for me.
https://raw.githubusercontent.com/ngosang/trackerslist/maste...
And paste the list in my torrent app. The above list updates daily or so, so just by visiting it you get the latest and greatest.
There - I said it, we need more GitHubs around the world, hosted in every country to curb censorship :)
Bittorrent is based on torrent files which describe a specific list of files. Is it really required to maintain them as a single pack/folder? For years I have an idea where one could just tell torrent client where their data storage is (with a flexible granularity ofc), so it could index it and share without any specific torrent file, simply by block hashes. I.e. if I have a file somewhere deep in the tree, and someone wants to download a block that corresponds to that file, or a part of it, it would be served without being a part of some torrent on my side. Why it is not a thing from the beginning?
For me the upside is that I could sort and categorize files (even install and forget) without removing/doubling them off the “Torrents/“ folder. I believe this could reduce maintenance burden, thus seeders will not give up on seeds only because there is too much to handle. And also provide vast deduplication and cross-torrent seeding. It’s like a distributed filesystem, where you autoshare inodes instead of prepared folders.
If what you want is to be able to send a very specific file to someone, then single-file torrents exist for this purpose already. There might be some manual work indeed, but that's only because no one really had the need before.
Private is not used for DRM, it's used for security on every private tracker.
The main problems are that:
1. torrents have different “piece sizes” (i.e. the torrent creator chooses a power-of-two size for the partitions of the torrent’s data that get their hashes stored in the torrent file); and
2. multi-file torrents produce “pieces” that are an arbitrary split of the concatenation of their files, as if they were actually a torrent of an archive file—just a virtual one synthesized on the spot by the torrent client, as if it was using FUSE to translate the filesystem it sees into a packfile to share.
The “solution” to problem #1, if you can call it that, is to walk your files and generate (log N) different torrents per file, for each piece size. Then—at least for single-file torrents—you could in theory build a tracker (and client-API-hooked search engine) that exposes “virtual torrents” for every possible combination of pieces (when fed this sort of... torrent-data overlapped Merkle tree.)
I don’t think there’s really a more clever solution to problem #2 than the one torrents already use, though. It really does make sense to pack files together into single large pieces, when files are small and numerous, because the metadata of sharing a single file on the network (or, to a lesser extent, of storing a single file separately on disk) has overhead, and so people don’t tend to like seeding tons of tiny files, preferring to instead seed archives of them. (Or, even worse, when a file is seen by most as useless except in the context of being a “resource file” of a particular application, people will only ever bother to seed it in the form of the installation archive of said application.)
To find a particular tiny file on the network, then, you need an index mapping the metadata (and/or hash) of the file you want, to the metadata and hash of an archive/packfile that contains it, and which is tracked by the network. Which is... the thing torrent search engines—usually built on the backs of torrent trackers—give us: the ability to plug in a filename, and find torrents that list that name in their manifests.
(IPFS doesn’t solve problem #2 either, if you’re curious. It just keeps everything as individual file hashes, and then allows you to retrieve directory manifests by their hashes—but you can’t predict a directory manifest hash from a file hash, in order to discover the directories that “make use of” a given file, and might therefore have a more-well-pinned packfile equivalent representation that includes that file.)
I think we could solve #2 if we had some sort of system of hashing—not necessarily cryptographic hashing—that enabled us to ask, in O(1) time, “given that the hash of a packfile of bytes of size Xsz is Xh, does the packfile contain a file whose size is Ysz≤Xsz but otherwise arbitrary, and whose hash is Yh; and if so, at what range within the packfile can we find the file?” (The “but otherwise arbitrary” part is important; we can’t just pre-chunk up the packfile and hash all possible chunks of it, like we can with #1, since the files in it might be in any arbitrary positions with any arbitrary lengths, potentially even straddling a piece boundary.)
If we had such a hash algorithm, then any file hash of this type that a tracker received in a request could be used as, essentially, a zero-knowledge proof of what packfiles contain that file (since you can just iterate all the packfiles you have and check whether they contain it, and use that to build an inverted index from segment hashes to packfile hashes.) But I don’t think a hashing scheme like this currently exists. (At least, in this use-case... YouTube’s ContentID algorithm is kind of an implementation of this for audio fingerprinting. Or maybe not; maybe it’s just pre-generating comparator fingerprints for every possible slice of each claimed audio track!)
Ed: seen your edit, will read later, thanks for your thoughts on this.
But most files we might want to find on the network aren’t actually left “unpacked” on disk by their seeders and only virtually packed at torrent creation time; but rather are sitting in existing archive files that the seeder wants to redistribute unchanged (usually because the archive is cryptographically signed, or is a functioning self-extracting installer, or has some other property it would lose if the extraction of the archive’s contents were given instead.)
In such cases, a naive index wouldn’t work; but you could just slap libarchive into torrent clients, allowing them to pull out the leaf-node hashes from files packed into an archive file at any depth (and also the branch-node hashes of any intermediate archives.)
In a sense, we’re talking about the same thing that virus scanners do, except the “signature database” is a produced and shared artifact of the network rather than a manually-curated set!
This requires every torrent file in existence to be re-created from the source data, though, which is probably unlikely at this point, unless someone is planning to write the equivalent of Archive.org’s Wayback Machine to spider all torrent trackers, download everything they’ve got, recreate the torrents, and seed them in the new format, which is hopefully backward-compatible with the old format in such a way that this seed shares peers with the existing seeds.
I also thought of a thing similar to libarchive - a format analyzer which could e.g. separate a movie from its subtitles and audio tracks for specific formats, enabling world-wide availability for heavy streams. Or sharing jpegs/mp3s with metadata changed to ones needs, but with main parts intact and thus still seedable. It’s not about really splitting a file in parts, but about detecting better block beginnings, sort of network-wide struct member alignment.
- https://torrentfreak.com/publicbt-tracker-set-to-patch-bitto...
- https://torrentfreak.com/hollywood-appeals-decision-not-to-s...
- https://torrentfreak.com/court-refuses-to-order-shutdown-of-...
This mode of operation is one of the fundamental modes of opentracker - http://erdgeist.org/arts/software/opentracker/ - which is the software pretty much all of the many open BitTorrent trackers run.
However, from what I read some time ago, it’s actually connected to Pirate Bay, and when Pirate Bay used to be down, OpenBitTorrent was also down.
This tracker started in 2009 and has gone offline twice since, returning from the ashes most recently in 2018.
It runs on opentracker software that anyone can download and run, but it would seem popularity and widespread use make a tracker more useful to the public.
It also bears the brunt of legal issues since it remains high-profile and subject to numerous court cases.
It has gone down a few orders of magnitude more times than that.
I'm not sure why this is really necessary given the existence of trackerless DHT torrents though. Maybe using a tracker makes it faster somehow?
Also, FTP servers? Are there public FTP server people operate? Why?
This is by design, but I agree: it would be nice to have at least a way to point users at a place where a common standard and open protocol would allow communications related to the torrent. An IRC channel named after the file hash maybe? In the old days of Napster and later the OpenNap network (anyone here used the Lopster client?) I discovered a lot about unknown rock bands just by looking at files shared by other people and asking them for more information. Those were the early days of multiuser downloads, when downloading a not so famous movie could take like two months (it happened to me) so pestering the poor user on the other side with continuous downloads wasn't an option. Being able to chat also gave the users the opportunity to help each other to solve networking problems or other technical issues. Having some way of communicating without altering the protocol (that is, external server) with the torrent containing just a field in a structure identifying a channel, could be an interesting improvement.
And responsibility, a requirement to keep the software up to date, and an increased attack surface.
> In BT there's none of that.
Not anymore I suppose because it used to be mandatory to stay on good terms with channel and server ops.
> Also, FTP servers? Are there public FTP server people operate? Why?
To save costs. You'd think an ISP in for example The Netherlands who host a mirror of (legal) .iso or .deb is doing this for fun but in reality they're saving themselves costs because customers don't have to download from further away (where they got less bandwidth and less good peering agreements than their local server).
You are getting peers from tracker in the response and it's more data than in announce, which means amplification attack is possible.
> announcing to it doesn't cause it to connect back to you.
UDP is connectionless protocol, which means it doesn't connect to anything, it just sends data.