Hacker News Reader

Comment by gruez | Hacker News Reader

I wonder how intelligently this handles piece download order. Torrent clients usually use rarest-first download algorithm, but if you've mounted a movie and you're playing it with VLC, you'd probably want it to download sequentially so there's as little time spent buffering as possible. You could download the pieces on demand (whatever piece the application is trying to read), but I suspect that will provide a suboptimal experience, especially if your download speed is barely above the bitrate of the movie.

View on HN

I have been using btfs for years so I can answer.

It can handle videos and other streamable formats if there are enough seeds: the connection is not usually a problem. The pieces are not downloaded strictly sequentially but using a sliding window[1], so if a piece is not immediately available btfs will not waste time and bandwidth waiting for it but just keep downloading other pieces nearby.

The problem is until enough pieces have been assembled, applications reading the file may literally hang: I guess not much software was made with slow I/O in mind.

[1]: https://github.com/johang/btfs/issues/7#issuecomment-1916443...

There are all kinds of reasons a filesystem could be slower to serve certain bytes: striping across multiple heterogeneous physical disks, network filesystems like nfs/cifs/btfs, even IOp starvation.

Video players can gracefully handle streaming media despite spotty connections; is there a reason that they can't pre-read a sliding window of bytes from a file on disk and plow on in case a given range of bytes is taking too long? They could default to wanting all the bytes, but have a setting that instructs it to treat the filesystem as if it were a network source.

> is there a reason that they can't pre-read a sliding window of bytes from a file on disk and plow on in case a given range of bytes is taking too long?

AFAIK vlc has that functionality. It allows you to buffer the video n seconds in advance.

Setting a modest buffer size helps but does not solve this entirely. For example, turning on the subtitles or switching audio track will usually hang the player for a while.

Is there any way to get it to immediately start downloading the files, then prioritize the bits requested by applications reading the file?

Also, very curious, how have you been using btfs? What have you been using it for?

I think this should be the default behavior: as soon as you mount the torrent btfs start fetching it like any other torrent client.

I use it mainly for streaming videos: btfs comes with a small python script "btplay <magnet>" that will mount the torrent and start your player, very handy. Another cool thing you can do is downloading and extracting a file simultaneously, just mount a torrent and call "unzip".

I wrote https://github.com/anacrolix/torrent specifically to address the on-demand and streaming requirements of some products I created. It also doesn't make assumptions about how you store the data as you stream, so there's no requirement that the files exist on disk as they're described in the metainfo, like other clients.

libtorrent has repeatedly ignored requests for sequential download because of the silly spec. There's a patch out there that quite easily enables it. It's a bit absurd to me because there's a very good use case for sequential download and it's not clear there are any actual side effects.

Doesn't sequential downloading degrade the swarm performance?

Yes, but only 2nd order...

If all users downloaded in order, then users downloading the file cannot exchange chunks with other downloaders.

That "exchange" is necessary to encourage users not to limit their upload speeds.

In today's world where the vast majority of users just use the default settings for their torrent client, it probably doesn't matter much.

Maybe, but probably less relevant for seeds that are not new. Also if people are streaming via torrents it's a net benefit if all peers have early bits not just random data. In short I do not have empirical evicidence but I do believe the decision to do so should not be made by a library and I also believe the intial assumption that random is better holds true today.

See my comment https://news.ycombinator.com/item?id=23580765 if you want an alternative.

Libtorrent-rasterbar has supported sequential downloading for a long time. Are you sure you're not thinking of the other libtorrent (libtorrent-rakshasa)?

I'm not sure clients actually download rarest-first. It's supposed to be random.

From The BitTorrent Protocol Specification, http://bittorrent.org/beps/bep_0003.html:

> Downloaders generally download pieces in random order, which does a reasonably good job of keeping them from having a strict subset or superset of the pieces of any of their peers.

However clients can certainly choose to download in whatever order they wish. Like Popcorn Time.

Reply on news.ycombinator.com