Comment by pzmarzly | Hacker News Reader

Comment by pzmarzly | Hacker News Reader

It is easy to make a torrent client, but very hard to make a good torrent client. A very good, or let's say "perfect", one, needs to support multiple transport protocols (TCP, "uTP" aka UDP, "WebTorrent" aka WebRTC), multiple discovery mechanisms (DHT, PEX, HTTP trackers, WebSocket trackers), multiple torrent formats (v1, v2, hybrid), should use the network optimally (max the speed without overloading the network - IIRC some clients measure average packet latency and if it starts going up, put some backpressure), resolve magnet URLs, set up port forwarding, reconfigure firewalls, offer API for *arr stack, be a good netizen (report stats correctly, send packets within the specs, do not spam - otherwise other clients will blocklist your in their code or config), implement many BEPs (mutable torrents are cool), be able to recover from interrupted state based on only the data that's on disk, have configurable downloading order (people want to start playing videos before they finish downloading, so you may want to e.g. download header and footer of each file first), and ideally detect duplicates between torrents (cross-seeding). And then there will be people throwing 2TB+ torrents at it (e.g. TLMC) to benchmark it, and saying your client is "literally unusable" if it doesn't handle it.

So, building a "perfect" torrent client from the ground up is a daunting task. But the "good" news is that nobody built such a "perfect" client just yet, so if you have some spare months of your time, you can take a shot at it. Or even better yet, open the issue tracker for one of popular clients or libraries, and add one of the missing features from the list above.

Few more challenges:

- seeding a million active torrents (this is entirely possible and needed but drives a lot of architecture)

- a UI for a million active torrents

- be nice to trackers (keep-alive, batched reports; again, architecture gets tricky, iirc libtorrent can't do that still)

- correct pipelining

- congestion control on uTP

- I/O scheduling/batching

- testing all of the above

It is quite a lot.

Looks like we have same experience with torrents :-)

Yeah. I was working on [0] for a while (I might get back to it when I have more time) and implemented a chunk of the protocol with an eye for a more scalable libtorrent replacement, and it's not entirely trivial. Even things like the piece picker get trickier at scale.

[0]: https://dgroshev.com/blog/torrentdyne-encryption/

My client anacrolix/torrent has most of that, including streaming, hybrid, v2, all trackers and protocols.

I don't do much port forwarding, it's too fiddly. I think I provide upnp out of the box and few other variants that were contributed.

I also have all the DHT extensions, like mutable torrents and get/put etc.

Large torrents do pop up occasionally, but it's been a long time since someone found a performance issue there but it used to happen every few months. Lots of optimisation.

GitHub.com/anacrolix/torrent

I know your library does a lot, I have been your GH Sponsor for months ;) Thanks for reminding me about it, I wanted to up the amount. And to thank you for working on such cool stuff and making the ecosystem better.

I missed the upnp support, thanks for correcting my beliefs.

Which client with WebUI would you recommend the most? I tried using exatorrent and distribyted, but was running into bugs.

Any clean room impl. is good news to me, but is the support for private torrent/tracker complete (https://github.com/anacrolix/torrent/issues/531)?

NB: especially after https://github.com/Luminarys/synapse died and Transmission "botched" its 4.x rewrite (currently on 3 with some backported patches).

another person still using transmission 3 here! no clue what i'm going to do if/when it stops compiling.

https://github.com/stefantalpalaru/transmission-og

Also, good disk I/O scheduler needed, or you can thrash even stripped NVMe disks, not to mention spinning rust.

Good multi threading without any global locks is a must or your UI/remote control become unresponsive on big number of active torrents.

Also, it needs daemon mode, which can be compiled & installed without any GUI libs (like QT, GTK, Waynald, libx11, anything like this) and fast responsive remote GUI for all platforms which allows to manage thousands of torrents simultaneously (I'm looking at you, <any-existing-torrent-client>). It could be Web UI, but for now I didn't se usable one. Only thing that semi-works is Transmission + Transmission-Remote-GUI written in Pascal. But it is effectively abadonware and works worse and worse with each Transmission update.

And be scriptable to allow automate change of torrent on tracker, for example (you need to detect it by "Torrent not registered" tracker-specific error answer and re-download torrent building URL from, for example, "comment" field of torrent, again in tracker-specific way).

> But the "good" news is that nobody built such a "perfect" client just yet,

Which of the qualities you listed are lacking from the currently most popular torrent clients (qBitTorrent, etc..)?

Personally, "being as small as uTorrent used to be", but clearly that's not a deal-breaker. (Then again, neither is like two thirds of the stuff GP mentioned!)

To be fair, circa 2010 all torrent clients got "good enough", so my wishlist above is just a wishlist (because torrents are cool and I like to see progress). I use (and sponsor) qBittorrent myself and have no plans to change.

I am using qBittorrent, and from the top of my head:

- WebTorrent and WebSocket patch for qBittorrent is ready but not merged (waiting on libtorrent's decision),

- cross-seeding support is poor (a separate "cross-seed" binary can be used to set up hardlinks to fool qBittorrent into cross-seeding, but it cannot detect duplicates on its own)

- when it comes to torrent management, there is no way to group torrents into groups with common settings (important if you use multiple private trackers) - people recommend having multiple installations of qBittorrent side by side

- when it comes to reconfiguring NAT and firewall, qBittorrent supports UPNP IGD protoocl, but I am not sure about NAT-PMP and PCP

- I have never seen qBittorrent connect to a single IPv6 peer - so I don't know if the support is there

- download order - you can choose "download in order" or "download rarest first". I dont think "download in order" downloads footers, so mp4 files won't work (IIRC mp4 store metadata in footer, mkv in header)

See my other comments, but my client fully supports Webtorrent and webtrackers.

Not much support for port forwarding.

IPv6 definitely works.

Download order is handled by providing readers directly into torrent data and using that for prioritization. So basically request what you need when you need it. No arbitrary list of algorithms.

> NAT-PMP

https://github.com/arvidn/libtorrent/blob/c31d90b59ffb3910b6...

>PCP

seems like vaporware

Isn't PCP the only protocol of these 3 that can reconfigure IPv6 firewall?

> NAT-PMP and PCP

qBittorrent (via libtorrent) supports NAT-PMP and PCP

> IPv6

qBittorrent supports IPv6

qbittorrent has "download first and last pieces first" (in addition to "download in order") so the footer is retrieved

> - when it comes to torrent management, there is no way to group torrents into groups with common settings (important if you use multiple private trackers) - people recommend having multiple installations of qBittorrent side by side

Probably easiest to do this via docker-qbittorrent-nox.

> - I have never seen qBittorrent connect to a single IPv6 peer - so I don't know if the support is there

The Linux ISOs (not a euphemism) I'm seeding probably get about a third of their peer connections via IPv6.

Last I tried it their hybrid torrent implementation was very broken and would fail half the time.

For me, the hardest part of building a good BT client (or any software, really) is getting the UX right.

These days, I use QB after uTorrent's downfall, but even after all these years, its UX still isn’t quite there.

qBittorrent + VueTorrent is not perfect, but it's hard to find anything better. If you haven't tried it, take a look.

https://github.com/VueTorrent/VueTorrent

https://tixati.com

Tixati is simple in the same way Deluge is simple. Because of this they both lack a lot of the quality of life things out of the box that qBittorrent provides.

As someone who stopped torrenting around 2015 or maybe 2016, what happened to uTorrent? Also RIP what.cd, I still am sad that it is gone.

Became adware

Alternatively, a minimal client with few features that can seed large numbers of torrents without crashing or freezing. Settings for how many it may seed at a time, prioritize that what needs seeding, ignore overly seeded things. Options to seed only blocks no one has. Remotely controlled.

I had a super seeder implementation that seeded over 400k torrents simultaneously using some fancy callbacks at certain points in the protocol.

If this is actually a common use case, I can resurrect it into a usable form for the public if there's interest or funding. https://github.com/anacrolix/torrent

https://www.reddit.com/r/DataHoarder/comments/wcusy1/torrent...

The datahoarders run multiple clients. It's a workable solution but far from ideal.

> but very hard to make a good torrent client

> So, building a "perfect" torrent client from the ground up is a daunting task.

Don’t you think you’re exaggerating a bit? It’s not daunting by any stretch. The feature set you described is fairly straightforward and something even a beginner developer could tackle without too much hassle.

I’m honestly kind of tired of seeing people act like doing anything these days is some impossible feat.

Amen.

It's not impossible, it's just hard to be better than libtorrent, otherwise a new client only makes sense as an exercise.

Arvid Norberg is hardly a "beginner developer", yet look how hard it is to make a performant implementation of uTP: https://github.com/arvidn/libtorrent/issues/3542

Why would it be an exaggerated claim?

Try it yourself, from scratch, and see how quickly you will finish something that will rival and surpass, say, qBittorrent.

None of the tasks is breaking new ground or is creatively difficult. But there's a metric ton of those tasks in order to make a truly good torrent client.

Given the state of software quality, it is remarkable that torrent clients are SO GOOD, given they are free!

There's only a handful of actually complete ones. A few it's surprising they even work, they're crazy complicated, and weird, but once they're stable they need very little maintenance. They're all also written in very different styles due to a mix of C/C++ and very high concurrency.

Yes, even qBitorrent "destroys" your other connection at maximum speeds.

You may also want both a CLI and a web access/management interface

All of that without encryption. What a meh

But shouldn’t be hard to do with modern facilities (async runtimes, streaming libraries etc.) I don’t enjoy many clients yet I use them.

It’s definitely a good problem with which to test a programming language. Many fail on parsing, or inefficient network interfaces, inefficient file interfaces, sha sum unusable without intrinsics, etc.

Reply on news.ycombinator.com