Distribyted: Torrent client with on-demand file downloading as a filesystem | Hacker News Reader

Distribyted: Torrent client with on-demand file downloading as a filesystem

github.com

75 pointsmastabadtomm26 comments

Somewhat off topic, but is anyone aware of a bittorrent proxy?

I regularly end up downloading torrents on a remote box, and then have to copy them to my local network. In an ideal world, I could run a torrent client at home and have it proxy though my vps so it doesn't leak my home IP.

The other way I can see it working is being able to add a torrent to both my vps and server at home, and have the torrent client at home only ever talk to the single remote box so I still end up with the torrent at home without leaking my home IP and without having to download the finished files as a second step.

I've heard decent things about put.io for the remote side of things, then using `rclone move` [1] to pull the content down at frequent intervals (put.io doesn't show the files till they're done downloading so no worries of pulling half files)

If you're not a fan of the full setup there you could look at just the rclone part and have your torrent client move completed downloads into the folder you pulling using rclone

[1]: https://rclone.org/commands/rclone_move/

I realise this isn't quite what you've asked for but the direct answer would be to set up either a VPN or a proxy yourself, of which I have no suggestions or recommendations software/host-wise

put.io has rclone support? I've been using aria in the past and it's not bad but not super convenient.

EDIT: https://help.put.io/en/articles/3480094-plex-rclone

Aye. This is a Plex-centric guide but you can ignore the plex part https://help.put.io/en/articles/3480094-plex-rclone

I'd personally go off-guide and use rclone move rather than mounting it just to avoid having to keep bumping up storage space on put.io

e: hah you already got it

Makes sense. I use plex, but I've only got ~100GB at put.io and it fills up fast (especially with those DS9 AI-upscaled seasons...)

I'm curious to hear if anyone has advise on the former as well, specifically which iptable rules to apply in order to ensure that both incoming and outgoing are routed properly.

The incoming is straight-forward using your load balancer/reverse proxy of choice, it's the outgoing that's tricky.

Or, as you say, have two torrent daemons. I have'nt done this myself, but what I have in mind, a script that:

* Syncs torrent/magnet to remote (scp or syncthing) to your vm's torrent daemon's incoming folder

* Produces a new .torrent file that is added to your local client, with only the single peer. Ensure that PEX and DHT are disabled.

For this use-case you needn't care as much about leaking, as long as you take care of the above.

Up to you, but the benefit of the bittorrent protocol is downloading pieces from multiple peers at once, and then seeding that content back out to support other peers accomplishing the first part. That makes its use in a two peer, download-only, setup kind of weird.

I could imagine that being a good setup if your home machine is already managing linux isos, and you want to download ... uh, your own linux iso ... from a VPS (i.e. a concession to the existing process, and not the BT protocol itself)

Do you mean a VPN?

But be warned, your home internet, VPS, and/or VPN are all most likly all strongly tied to you. That is because you paid for them.

A VPS where you are the only user is a 1:1 mapping to you.

It's not just for the sake of hiding my IP - I torrent a lot of linux distros. It's just nice to be able to seed at the 200mbit of my VPS than with the 20mbit upstream from the home connection - but this is a very good point.

I also use the same setup. I made a script using python transmissionrpc which watch the torrent pieces being downloaded and send them with sftp as soon as they are available.

Then on the receiving side it just creates a sparse file and fill in the pieces using dd (so no dependencies on receiving side).

The sending speed was not very good. It could probably be improved by doing multiple transfers in parallel and using a less hacky way to transfers the data.

I got something similar, but without a remote box (or a remote box that I control, rather).

I made a docker container that has the torrent client and openvpn installed in it. All torrent traffic bounces off the VPN, and uses the VPN IP but I am still able to access the torrent client interface via the direct IP

I like xdcc because you dont need to upload

I was actually sketching out an idea like this. Except that each file would have a header with a list of entries encrypted using public keys. When you found an entry it that you could decrypt, it would tell you how to reconstruct the desired data from rest of the file. The best part of it is that you have no idea what any of the other entries could produce. Your cat gif could contain a whistle blower's document.

You may be interested in Freenet. Data is distributed across nodes but the nodes don't know what they have. An individual file isn't necessarily entirely on one node, it could (and most likely is) distributed across many nodes. In order to retrieve data you need to know the key for that data. At that point you can start requesting the appropriate blocks from the network and reconstitute the file(s).

Super cool. It would be really awesome to get this in the web browser as well. I would love to have access to my music/movie collection on anything with a browser. You also wouldn't need to worry about setting up DNS/Public IP for the host.

github.com/anacrolix/torrent supports WebTorrent so it shouldn't be that hard.

Not over bit-torrent but this exists: https://syncthing.net/

Syncthing does not work in the browser and does not provide on-demand access to files. It syncs directories.

Not yet, but it could.

On-demand access to files is a completely different workflow than syncing. It would also clash with the name, and require a change in technologies to run in the browser, as well as transport protocols.

I don't agree. Dropbox syncs files across all of my devices, but I can also get files out using a web interface and mobile device apps (which, coincidently, is why I don't use syncthing). Regardless of the name, I think on demand access would be complementary to it's primary purpose (syncing), and serving up the content over http would be trivial (as well as over bittorrent/webtorrent, considering existing library support [1] [2]). At it's core, syncthing is an object distribution orchestration system, therefore other transports besides native syncing should always be considered (imho).

[1] https://news.ycombinator.com/item?id=23818659 (HN: Libtorrent adds support for the WebTorrent protocol)

[2] https://www.bittorrent.org/beps/bep_0019.html (WebSeed - HTTP/FTP Seeding (GetRight style))

You know what, if you are willing to spend this much time proving that you are "not technically wrong", I will give it to you. Sure, everything can be turned into anything, given enough effort. And Syncthing does show similarities if you squint really hard.

It doesn't change the fact that the "have access to my music/movie collection on anything with a browser" mentioned by the grand-parent is not what Syncthing does, and is not even close to what it does now or what the team is interested in implementing [1]. While it is a great piece of software (that I personally use), this recommendation was misguiding.

Have a good rest of your week.

[1]: https://github.com/syncthing/syncthing/issues/6753#issuecomm...

The constant "Out of Sync" errors in Syncthing make me think it's not so great at it's first job.

Is this like IPFS?

People have long described it as "Bittorrent inside a filesystem."

It appears to just be a torrent client with an smart FUSE capability.

Is this read only? Can files be edited?

Sounds a bit like PiperNet

Reply on news.ycombinator.com