BTFS – mount any .torrent file or magnet link as directory | Hacker News Reader

BTFS – mount any .torrent file or magnet link as directory

github.com

510 pointscab404121 comments

This is effectively what IPFS is, but IPFS takes it to the next level by using the equivalent of torrent files for subdirectories within the mounted torrent files, so you can "traverse" entire networks of the pretty-much-a-torrent files without first having to download and mount them. And like, you could implement similar functionality here by having torrent files that contain files named .torrent have those files be treated as directories, but at that point IPFS is going to be much more feature complete; even like, if you have some giant file, on IPFS that might be represented by trees of hashes as opposed to merely a giant array of hashes so you can incrementally and efficiently access the parts you need. So like, while this is awesome work, if you find it inspiring, maybe channel that inspiration into learning about IPFS and extending that ecosystem, rather than doubling down on trying to build on torrent files.

Usually, an upvote should suffice, but this is the best ELIF of IPFS that I've read.

Are there any design changes that could create organic adoption of IPFS?

Or, conversely, what design decisions do you think have hindered adoption?

I've had a breif look at IPFS, but not enough to say anything with any amount of confidence. That said, the thing that immediately put me off was that everything seemed to want to be browser based, like starting some service that makes a local web server that acts as a GUI - there was even a chrome extension that tried to make this easy for you.

A feature that might exist, but I haven't discovered, that would make IPFS more appealing to me, would be a way to mount the IPFS like it were an FTP server, and navigate it with whatever file manager you prefer and not have to worry about managing the backend stuff.

That is to say, I'd like BTFS for IPFS.

It's built into ipfs already: ipfs mount

Given the other reply to your comment was someone who seriously thinks IPFS's core feature--being able to be used as a filesystem--doesn't exist, I guess I'll have to go with "better onboarding documentation" ;P.

The lack of standard dns glue, i.e. a domain which turns to local client if you have it, web gateway like ipfs.io otherwise.

And for ones who are after a product which can already do all those written above can go for Sia Skynet instead of waiting IPFS.

What are the current best practices for hosting dynamically updated content on IPFS? Something like a blog that gets updated once a day or a Hacker News clone that gets multiple updates per minute. Last time I checked it was DNS, has something changed sice then?

Something that's as dynamic as HN seems to me like a bad fit for IPFS. A blog seems fine for it; I make my own blog accessible through IPFS.

DNS seems like the best and currently most popular option for letting you have an address with an update-able IPFS link. The nice thing about using DNS is that you can use the same address for people accessing via IPFS and people accessing via standard browsers.

HN is too frequently updated, but for blogs or personal web pages it's okay. You can either use IPNS or update a DNS record each time you update your content. I chose the latter as years of experiences proved it more robust than IPNS (also, my registrar provides a very simple API that I can use to automatically update the appropriate DNS record, so this is done by the same Python script that update my IPFSite).

The downside is that hosting an IPFS node is quite heavy on the network. I have a reasonably good connection at home (a bit less than 1MB/s up, a bit more than 10MB/s down) and if I run the IPFS daemon locally I can't really use my internet connection for anything else. I used to rent a little VPS for that, but now I use the free-tier of temporal.cloud and it is sufficient for my usage.

¿Por qué no los dos?

How is it "doubling down", they could exist side by side. Your comment kind-of shows the mentality I've seen with the IPFS project before. The IPFS project has doubled down on it's effort of being more "advanced", "innovative" and "efficient" than BitTorrent, losing what actually makes people like torrents and distributed systems.

Unfortunately right now the IPFS project is totally ignoring that it's effectively nothing and takes nothing to the next level. I think what I mentioned before, is the root cause. Zero ounce of compatibility with the existing BT swarm. It is simply a massive amount of wasted potential to ignore the thousands of terabytes of content the torrent system hosts, the work and services provided and spent for the ecosystem. So much could be done that would strengthen both systems at the same time. Just from the IPFS point of view we could right now have embedded resources from torrents, tens of different clients available that could just be CDNs, existing seedboxes strengthening both swarms and probably much more.

That would be the mix of torrents and web people want, not what IPFS is currently. IPFS is basically reinventing the wheel, but throwing out the round part.

Maybe the IPFS project should channel more work on building next to giants rather than doubling down and throwing away everything BT has to offer?

To be clear, I am not involved in the IPFS project and in fact don't like key aspects of the project and think it is doomed to long-term failure when it gets replaced by something better.

That said, the core and pretty fundamental limitation with BitTorrent, however, is that the swarms exist "per torrent", which kind of makes a non-starter for a general purpose filesystem?

As such, I don't quite disagree with what you'd like to see done, but that work should go in IPFS... like, trying to extend the way BitTorrent works "up" to something that works for this use case is going to involve a much more fundamental set of low-level changes than even the WebTorrent shift (which is where I'd argue "if only BitTorrent understood the value of working with existing browser-based systems and opening up powerful use cases they would understand that not doing this is like throwing away the round part of a wheel... etc." ;P WebTorrent is the future of BitTorrent as far as I'm concerned, and anything failing to embrace that is already kind of doomed?...) than the other direction, which is really just "go into IPFS and add a little adapter that lets it work with torrent files (at the fundamental loss of efficiency and awkward lack of resource sharing that such usage implies).

The primary difference I see here is that IPFS is rather centralized in terms of development, it is much much easier for them to write a BT compat layer of some sort than it is for the BT community. A core set of BT is really rather ossified thanks to the amount of implementations out there. Just as an example there are quite a few BEPs out there that implement some features that IPFS has, unfortunately those features haven't gotten any mainstream adoption so far.

> That said, the core and pretty fundamental limitation with BitTorrent, however, is that the swarms exist "per torrent", which kind of makes a non-starter for a general purpose filesystem?

Well, ipfs nodes also tend to pin only certain content. I've seen IPFS links stop working mentioned here on HN even. I see no functional difference in that sense. Content is kept available by those willing to spend resources storing and uploading the specific content.

To address the claimed performance/efficiency problems, IPFS peers could share the same files using IPFS itself, but still also seed them for BT clients.

I totally get though that it wouldn't be super easy to implement, but IMHO the increase in IPFS usefulness, size and resiliency thanks to the integration seems to outweigh the downsides I can see.

Though a while ago I did try sharing the same set of files on two systems by running the two in parallel. It kind-of worked but manually updating the file locations in the torrent client and having an overview became too annoying, it should be built-in. E.g. IPFS sees `x.torrent` and `x`, then assumes `x` is downloadable from BT using that torrent file.

> WebTorrent is the future of BitTorrent as far as I'm concerned, and anything failing to embrace that is already kind of doomed?

In some use-cases, absolutely.

The source code is far smaller that I would've expected, tiny in fact - no seriously, its probably <2000 lines of actual code and looks extremely simple to audit and fix bugs. It seems the only real external dependency is libtorrent.

Honestly, if your not already amazed if not at the simplicity of this project, than you should be by the simplicity of fuse filesystems and the capabilities it exposes.

You are right, you encouraged me to look at it and the actual interface into fuse is here whic is at least understandable: https://github.com/johang/btfs/blob/master/src/btfs.cc#L968

If you think that's excellent and are interested in fuse, you're probably intereted in 9front which takes the concept to another level. Ftpfs does what it says, but webfs handles http and forms and the like, upas/fs can show your imap inbox, it's splendid and simple and allows your tooling to be the familiar nix-like utilities like awk and sed. Even capture your screenshot by catting /dev/screen to a file

Which lib/approach would you recommend if i need run a remote batch process using local files?

For example, on the local system:

remote-execute gcc localfile.c -o localfile.o

And on the remote system, there is a gcc and a remote agent which receives and retransmits the files

SSH for job running and management, NFS for filesystem access. Have remote systems mount the local file system. If NFS isn’t available, have the remote job pull files over rsync from your local system, then rsync back the resulting output.

Ugh! Thanks but no thanks

I dont want to mount and expose nfs, or allow a full ssh login. This will be a custom agent/daemon on both ends

I wouldn't recommend NFS over the open internet but there's a few options based around SFTP (rsync, scp, sshfs) and being based on SFTP means they can run without granting that user full SSH login access while still taking advantage of the security benefits that SSH brings.

For job execution you could write your own agent but doing likely wouldn't be any more secure than SSH. Just make sure you have disabled password logins (use keys instead) and fail2ban or equivalent running to auto blacklist attacks. You could probably use Chef or SaltStack if really wanted to avoid a remote shell but if you're not already running config management then you have to ask yourself if you're over-engineering a solution.

An alternative solution would be to run an OpenVPN tunnel and then you can SSH to your hearts content. But even here, unless you have multiple machines you want to connect to, I can't help thinking you're just making life harder for yourself without getting any realistic gains.

This is all based on the very high level spec provided so I accept there might be some currently undisclosed detail that renders the above suggestions moot.

NFS works great over the open Internet, as long as you do it through a secure tunnel. I've been doing this for years as a way of increasing the size of the available storage in a VPS.

You run NFS within which secure tunneler?

Just use Wireguard, tbh.

It's really no different to running any other service over VPN or SSH tunnel. It does work well with NFSv3 but never tried with NFSv4.

I've used both vtun and WireGuard for remote NFS. On a good day, I get 100MB/s to the NFS filesystem on my San Jose AWS instance (from Vegas) via WireGuard. Note: That was before CoVid-19. CenturyLink/Quest has since (stealthily) throttled my bandwidth down from 1Gbps to ~750Mbps.

You would use rcpu(1) which essentially imports the remote cpu. In other words the remote processor "sees" the local file structure/namespace.

9front is full of "free carrots". If you want a remote vpn, just import another servers /net ontop of your process namespace and you're all set.

9front looks so very interesting. Thanks

It also stays at "looks very interesting", it's not a usable OS on a normal machine for everyday tasks.

It's only 2000 lines because it's in C++. I wrote a simple Fuse system to mount zip and rar files (read-only like this) in <300 lines of Python. Fuse is pretty easy to make useful things with.

So did I: https://www.stavros.io/posts/python-fuse-filesystem/

This is cool. I can mount most of the Internet Archive as a directory with this. Would love this to grow into what your consumer cloud storage services are but for torrents (a global file system you can mount locally, using torrent data as inodes, with visual representations for availability status similar to the Dropbox green check boxes). Perhaps even with support for web seeding from a storage system of last resort (Backblaze, S3, etc).

I know it's not good spirit, but mounting torrents into the filesystem has been done a few times already, there are many projects that do that already.

Also, it's not really advertised, but S3 can act as a torrent node: https://docs.aws.amazon.com/AmazonS3/latest/dev/S3Torrent.ht...

Sounds like they've dropped support for newer regions, though. Which is a shame. Seems like a decent tool.

The cool part is you can have S3 serve a torrent, and then disable access to the object with permissions once it reaches an adequate seed level. You’re no longer charged for outbound data, but S3 will continue to act as a tracker for the swarm.

Wow, that was very unexpected.

Just checked Amazon's S3 page linked to by the PP.

Question - how do the following "Notes" that Amazon has on it's page affect that?

"Amazon S3 does not support the BitTorrent protocol in AWS Regions launched after May 30, 2016."

"You can only get a torrent file for objects that are less than 5 GBs in size."

Likely just can only seed/torrent via older s3 regions is all.

Blessed be thy us-east-1.

FWIW the S3 BitTorrent support is very primitive and limited. It hasn't been updated since 2006 from what I can tell.

While you're here I have to thank you for your go torrent library, it's really nice !

I have never used S3's torrent capability, but at this point bittorrent is pretty much "done" -- there's not much to add to make the protocol better without breaking compatibility, which means there's not much support to maintain a working software.

But I'm not sure about it, maybe they don't participate in the DHT or don't use PEX, so that could be something they can improve.

Well,

I wrote a device driver for Windows 8 that would mount a .torrent file as a virtual disk way back in 2011. We pitched the intellectual property around to several large companies. The only interest we had in the technology was video game companies... where we could mark file pieces as 'high priority' and allow the player to play the game even when the game disk was partially downloaded.

The IP was sold to a company developing their next-generation console video game system as part of a 24 million dollar package and never saw the light of day. I suspect what they really wanted was the associated patents.

I wonder how intelligently this handles piece download order. Torrent clients usually use rarest-first download algorithm, but if you've mounted a movie and you're playing it with VLC, you'd probably want it to download sequentially so there's as little time spent buffering as possible. You could download the pieces on demand (whatever piece the application is trying to read), but I suspect that will provide a suboptimal experience, especially if your download speed is barely above the bitrate of the movie.

I'm not sure clients actually download rarest-first. It's supposed to be random.

From The BitTorrent Protocol Specification, http://bittorrent.org/beps/bep_0003.html:

> Downloaders generally download pieces in random order, which does a reasonably good job of keeping them from having a strict subset or superset of the pieces of any of their peers.

However clients can certainly choose to download in whatever order they wish. Like Popcorn Time.

libtorrent has repeatedly ignored requests for sequential download because of the silly spec. There's a patch out there that quite easily enables it. It's a bit absurd to me because there's a very good use case for sequential download and it's not clear there are any actual side effects.

Doesn't sequential downloading degrade the swarm performance?

Yes, but only 2nd order...

If all users downloaded in order, then users downloading the file cannot exchange chunks with other downloaders.

That "exchange" is necessary to encourage users not to limit their upload speeds.

In today's world where the vast majority of users just use the default settings for their torrent client, it probably doesn't matter much.

Maybe, but probably less relevant for seeds that are not new. Also if people are streaming via torrents it's a net benefit if all peers have early bits not just random data. In short I do not have empirical evicidence but I do believe the decision to do so should not be made by a library and I also believe the intial assumption that random is better holds true today.

See my comment https://news.ycombinator.com/item?id=23580765 if you want an alternative.

Libtorrent-rasterbar has supported sequential downloading for a long time. Are you sure you're not thinking of the other libtorrent (libtorrent-rakshasa)?

I have been using btfs for years so I can answer.

It can handle videos and other streamable formats if there are enough seeds: the connection is not usually a problem. The pieces are not downloaded strictly sequentially but using a sliding window[1], so if a piece is not immediately available btfs will not waste time and bandwidth waiting for it but just keep downloading other pieces nearby.

The problem is until enough pieces have been assembled, applications reading the file may literally hang: I guess not much software was made with slow I/O in mind.

[1]: https://github.com/johang/btfs/issues/7#issuecomment-1916443...

There are all kinds of reasons a filesystem could be slower to serve certain bytes: striping across multiple heterogeneous physical disks, network filesystems like nfs/cifs/btfs, even IOp starvation.

Video players can gracefully handle streaming media despite spotty connections; is there a reason that they can't pre-read a sliding window of bytes from a file on disk and plow on in case a given range of bytes is taking too long? They could default to wanting all the bytes, but have a setting that instructs it to treat the filesystem as if it were a network source.

> is there a reason that they can't pre-read a sliding window of bytes from a file on disk and plow on in case a given range of bytes is taking too long?

AFAIK vlc has that functionality. It allows you to buffer the video n seconds in advance.

Setting a modest buffer size helps but does not solve this entirely. For example, turning on the subtitles or switching audio track will usually hang the player for a while.

Is there any way to get it to immediately start downloading the files, then prioritize the bits requested by applications reading the file?

Also, very curious, how have you been using btfs? What have you been using it for?

I think this should be the default behavior: as soon as you mount the torrent btfs start fetching it like any other torrent client.

I use it mainly for streaming videos: btfs comes with a small python script "btplay <magnet>" that will mount the torrent and start your player, very handy. Another cool thing you can do is downloading and extracting a file simultaneously, just mount a torrent and call "unzip".

I wrote https://github.com/anacrolix/torrent specifically to address the on-demand and streaming requirements of some products I created. It also doesn't make assumptions about how you store the data as you stream, so there's no requirement that the files exist on disk as they're described in the metainfo, like other clients.

Can someone explain why I might use this instead of downloading a select group of files using a torrent file/magnet link? Is the advantage that it selectively downloads files on an as-needed basis? Or, is the product niche somehow related to the fact that the torrent file/magnet link is mounted as a filesystem, instead of only as a collection of files and folders?

The latter - it lets you navigate the file structure of a torrent as well as read (stream) file content using any of your usual applications, without having to download the files first.

Streaming. I thought about this in the interfile sense, but not the intrafile sense. This makes sense.

Okay. I'm afraid to ask. What happens in the case of a) a torrent with no seeders or b) Has seeders with enough data to get a directory listing but either the seeders "drop off" afterwards or are incomplete

Other experiences with network filesystems (CIFS/NFS) on multiple platforms sometimes have things getting hairy if the connection to the server drops while the filesystem is mounted or in use

Hi,

A quick glance at https://github.com/johang/btfs/blob/master/src/btfs.cc on line 190 reveals that no seeders would result in an infinite wait state: pthread_cond_wait(&signal_cond, &lock);

The author could mitigate this by automatically failing any reads where there are zero seeders and/or if the peer piece map does not contain the block.

I would be way more interested in this project if it downloaded normally in the background immediately after the torrent is added, and then downloaded specific bits as programs requested them. I haven't looked into the code base for this exactly but it says nothing about it in the Readme.

Otherwise, like you're describing, you could add a file and then go to read it a week later and be met with no seeders and thus no file when it could have been grabbed somewhere in between.

But maybe that extends beyond the capabilities of a file system?

This is a brilliant concept of a tool. I’ve never worked with monorepos before, but something like this with rw capacity would be an absolutely amazing pipeline, if every developer in the office and beyond are full nodes in the system.

Torrents can't write though, or are you referring to a shared filesystem like NFS?

Let's create a chain of immutable blocks with back pointers on top of this to create new versions of the data. Maybe even call it a... blockchain.

Check out hyperdrive (https://github.com/hypercore-protocol/hyperdrive), specifically the daemon. Torrents are notorious for being read-only, but hyperdrive builds on bittorrent ideas and is making read/write shared spaces a breeze

See https://github.com/anacrolix/torrent/tree/master/fs and https://github.com/anacrolix/torrent/blob/master/cmd/torrent... for a Go FUSE implementation of this. (Looks like it was started in early 2014.)

Neat idea, but it seems like abuse of the torrent protocol. If peers don't grab chunks randomly then it partially defeats the purpose of bittorrent.

Streaming services have shown that BitTorrent is practical for streaming in some setup too

Every torrent client I've used recently has an "enable sequential download" option. The only torrents this would realistically affect are maybe ones where the seeder may go offline when a bunch of clients have been doing sequential downloads and no one has the final parts of the content.

Which would be the same as no one would have those random parts.

BTFS with mutable torrents would make a nice piracy suite

And with the increase of Subscription fatigue, torrent may possibly make a come back.

Mutable "torrents" with fuse access exists, it's called hyperdrive (https://github.com/hypercore-protocol/hyperdrive). See the daemon in particular (https://github.com/hypercore-protocol/hyperdrive-daemon)

No. Hyperdrive is not like the official BEP-conform "mutable DHT item"-based updatable torrents.

They essentially say "download the most recent version signed by this pubkey", which ia notably distinct from hyperdrive's append-only log.

"download the most recent version signed by this pubkey" is literally how hyperdrive works.

The append-only log is hypercore, one of the building blocks of hyperdrive. Hyperdrive is a layer above and gives you a view of how the archive is at any point in time, with preferential access to the latest version.

TBF what I understood from the request of "mutable torrents" wasn't the literal feature, but a way to have decentralized nodes exchanging updatable content. Bittorent with the Mutable Torrents BEP does work, but it's nowhere near as developed as what hyperdrive is today. And as sad as it may sound I don't think it will ever be.

Yeah. My point was, that BEP 46[0] doesn't have any archive/log functionality.

[0]: https://www.bittorrent.org/beps/bep_0046.html

Isn't that just IPFS? And forget piracy; it's great for ready access to any data that you want mirrored to a lot of servers at once - imagine if you mounted /var/cache/yum over BTFS:) (...maybe with an overlay filesystem in the middle; I forget the internals of yum's cache...)

There are numerous tools available to support using torrents for OS and package updates across fleets of servers.

I mean, there's satellite and such, but do any use p2p to distribute files?

https://news.ycombinator.com/item?id=17303055

https://github.com/alibaba/Dragonfly

>At Alibaba, the system transfers 2 billion times and distributes 3.4PB of data every month, it has become one of the most important piece of infrastructure at Alibaba. The reliability is up to 99.9999%.

https://wiki.debian.org/DebTorrent

https://directory.fsf.org/wiki/Apt-torrent

https://docs.datproject.org/

Tools[1] like this have existed for at least 6 years. Don't know if the older tools still work though.

[1] https://github.com/mafintosh/torrent-mount

If you make your own torrent files, how to prevent it accidentally share with the world? Is there anything in the torrent files that can prevent this? (Program and configuration can make mistake.)

Yes, there are BEPs that describe not announcing or accepting connections in certain circumstances. There are also peer discovery mechanisms that are designed to work on local networks. Of course none of this matters if you mess up (and someone externally cares enough to take your data), or if your client doesn't support any of this stuff.

Why not torrentfs? What are the needs for obfuscation or association with btrfs ('better try remaking fs'? )

“Why this project updated yesterday and not the project last updated 3 years ago?”

Unless you mean a different one since you didn’t link source

Maybe https://github.com/anacrolix/torrent/tree/master/cmd/torrent...?

This looks pretty awesome. Does it work with private trackers?

Err. Why would it not?

edit: more constructively, this seems to just use libtorrent in the backend, which supports the 'private' flag https://www.libtorrent.org/features.html . Though even if you didn't, it would still 'work' in the sense of being able to download chunks.

It also depends on which client ID it exposes and if that one is whitelisted on trackers. But that's not really a technology problem.

That answer probably lies more so with the tracker's software and approved clients than this project's implementation.

Cool but can't really see any practical use of this...

This is freaking awesome!!

Wow, what a great concept.

Great stuff, thanks!

Well that is neat, albeit an unfortunate name (1-char off from BTRFS). Wonder what performance is like - does it try to preemptively download? Cache? I would expect it to take a while to run `cp ~/mnt_btfs/big_file ~/Downloads/`, but how about `find ~/mnt_btfs`? If it's performant, it could be really nifty:)

very unfortunate indeed as there's another project with this name: https://github.com/TRON-US/go-btfs

I dont know whether the maintainer is a supporter of crypto (as in tokens) but having a common name with a Tron product (or cryptocoins in general) can backfire.

It looks like the BTFS in the OP was the "original", with its first commit in July of 2015. Could be wrong, because the GitHub interface doesn't make it easy to get to the beginning of the 13k commits for go-btfs.

9d0e6a7ffb5deea2c8c8e555d7bf6bcab6fdc6ac 2014-06-26 Big things have small beginnings.

(I have an alias specifically for finding that out)

Personnally, I don't care about any of the crytpo-mining-coining and I think the two are sufficiently distinct.

Thanks for digging in.

That's when go-ipfs was started - see my other comment for more info

Yes, OP's btfs was definitely first - in Tron's git history you'll also see that it's a fork of go-ipfs, so the vast majority of commits is from IPFS people.

However it's good to see that Tron are no longer trying to hide the fact it's a fork. For a long time they didn't provide the source code of their btfs.

go-btfs first commit: Thu Jun 26 00:38:48 2014 -0700

johang/btfs first commit: Sun Jul 26 10:20:49 2015 +0200

That first commit is to the repo it is forked from, which was go-ipfs... so it didn't get the name until the fork at a later time.

My major pet peeve of Github: time travel is too hard.

At least you can always clone and then just ask git...

EDIT: Actually, since this was a fork, it would probably be a pain to hunt down, even with a local checkout. But it usually works.

What's Tron and why should I care about them? :)

a brilliant sf movie from the early 80s, should definitively watch it :)

There's also an amazing full length Daft Punk music video that sometimes is considered a sequel to the 80s film, but you'll likely want to watch the spin-off cartoon for a lot of the necessary plot.

In the plot of the cartoon the titular character (program) Tron is assumed dead, and then corrupted by a corrupt system. I can only assume this is why his name is being bad mouthed by some sort of crypto-currency scheme, as Tron was a good program, a security program built to defend the users and would never stand silent on for-profit exploitation of users.

The TRON Project is also a 1980-1990s project in Japan to develop IoT infrastructure. They even made their own CPUs and developed 3 different flavours of Operating Systems: - an embedded real-time kernel ITRON - an soft real time single user desktop OS called BTRON - CTRON a server operating systems with real time guarantees for packet/phone line switching

In addition they also developed an true international character encoding before Unicode. It wasn't ASCII backwards compatible which is why it wasn't picked by Unicode committee.

A good resource to learn about it is here: http://tronweb.super-nova.co.jp/homepage.html

I honestly thought BTRFS had added this as a feature until I read the linked page.

I'm on such a BTRFS kick right now that I reflexively up voted. Oops. Cool project though; I regret nothing!

Reply on news.ycombinator.com