WebTorrent Desktop – Open source streaming torrent client | Hacker News Reader

WebTorrent Desktop – Open source streaming torrent client

webtorrent.io

337 pointsfeross80 comments

This is what Feross and I have been working on the past few weeks. It's been a lot of fun. Here are a few of the things I learned:

* Electron is surprisingly nice to work with. It is simple, pragmatic, and thorough. System tray integrations, desktop notifications, launcher progress bar in OSX, installers for all three platforms, seamless auto updaters for Mac and Windows, a crash reporter, and so on --- all built in. Many of the little things that an app needs to feel native and professional have clean cross-platform APIs.

* Shout out to Cheng Zhao, aka zcbenz. He's incredibly prolific and seems to be nearly single-handedly developing the Electron project. Every single page of Electron docs or code I've looked at on Github, the latest commit on it is zcbenz.

* Peer-to-peer is moving very fast. WebRTC support in browsers was flaky and buggy until very recently. Even today, Chrome has a few big bugs left ( eg https://bugs.chromium.org/p/chromium/issues/detail?id=392651 ) and WebRTC is not yet supported in Web Workers ( https://bugs.chromium.org/p/chromium/issues/detail?id=302019 ). But things are improving quickly. Edge is getting Data Channel support soon. Safari is almost certainly gonna get WebRTC support as well -- it's already in WebKit, just not enabled in Safari yet.

Tons of cool new uses of the web are just at the edge of becoming possible.

I'm really excited about the way WebTorrent Desktop bridges the WebTorrent and BitTorrent networks. If enough people end up running it, BitTorrent in the browser will really happen. Projects like the Internet Archive can make huge files available for immediate streaming on their website, without paying for bandwidth. It's going to be interesting times.

Just wondering, because it is not explicitly mentioned in the docs, does it only stream or does it also seed the downloaded chunks?

It seeds, like all good torrent clients.

Yes. All torrent clients I know, including WebTorrent Desktop, seed while downloading. WebTorrent Desktop keeps seeding after you've finished downloading a file, and keeps running in the system tray after you close the window unless you explicitly tell it to quit.

We want to be good citizens and keep the swarms healthy.

Absolutely. I'm just asking because in certain jurisdictions seeding (illegal content) will lead to huge fines, and not mentioning this on the homepage might lead people to believe that they can safely use this software. This is especially worrying since "streaming" is oftentimes understood as a read-only operation which cannot be pursued by law enforcement.

The general rule of thumb that seems to keep you under the radar is to limit your upload rate. Most clients have an easy option to do this - although apparently this is non-trivial in this client[1]?

[1] https://github.com/feross/webtorrent/issues/163

I didn't realize zcbenz was involved. He was also a big contributor to node-webkit.

One more:

* It's surprisingly fun and liberating working on a completely noncommercial project.

I have no problem at all with commercial software -- especially if it's free of ads and "microtransactions" and is simply sold to the user for a clear up-front price.

That said, it's fun just shipping something that anybody can work on, with no boss, no formal user requirements, no worries about whether the app will sell or whether the client will pay you.

The only goal here is to make something we're proud of and which users enjoy.

And... one more:

* It's really fun to share project ownership with all the contributors.

From CONTRIBUTING.md:

> Individuals making significant and valuable contributions are given commit-access to the project to contribute as they see fit. This project is more like an open wiki than a standard guarded open source project. (https://github.com/feross/webtorrent-desktop/blob/master/CON...)

The first time I witnessed this in action was on the LevelUp project (https://github.com/Level/levelup). Since the days of the iojs fork, Node.js itself has adopted the same model.

Sharing commit access liberally is awesome for project health, because:

- The project can move forward even when the project owner is busy.

- It empowers contributors. It makes them feel like owners, because, well, they are. They end up contributing a lot more code.

- In the worst cast, it's Git, so bad commits can be backed out. But I've NEVER had to do this, or ever seen anyone else need to do it.

hey there! One feature that XBMC (now called Kodi) had since 2004 but I haven't seen anywhere else, is the ability to play the contents of rar files.

So you could download movies and never have to bother unzipping them.

It worked for multipart rars too.

The lore is that they implemented a custom rar algorithm but anyway, that was 14 years ago and its an open source project so I should see this somewhere else by now!

Since you guys are doing something over the torrent network where these kind of video releases are common (multipart rar, reminiscent of the release group's newsgroup roots), could you implement this functionality too:

When streaming, discover the sequence of the rar files in the torrent, prioritize the file downloads by rar number, and play the video file within?

This is an incredibly niche feature today because no releases come packaged in archives. Furthermore, to my knowledge you can't really "stream" archives.

They still do come packaged directly from scene releases. Smaller private torrent trackers have these releases. Public trackers usually contain just the media files so people don't have to unzip. But I agree, playing from archives would be a great feature.

I'd say the majority still are split into rars for the initial scene release but by the time it comes to the public trackers, and even many private trackers, someone has already unrared it for you.

I've implemented the same thing in a Usenet/NZB streaming project a few years ago - it's actually trivial since the multi-part rars are uncompressed, so the raw bytes of inner-content are there ready to be read. Just have to handle RAR headers/trailers. Seeking worked too since Usenet posts were commonly made up of 700kb posts, so if a seek occurred, you prioritize retrieving the required posts where the user seeked to, to fill the media player buffer. Code has never been released for obvious reasons :)

multipart rars are uncompressed? this whole time? my whole world is shattered right now

A bit of an aside, but for anyone who thinks this sort of thing is technically cool, Free Download Manager will let you look inside a remote ZIP file (of any size) and let you pick the files you want to download. Relatively simple to implement, but cute. It's the only open-source download manager offering this, I think.

(http://freedownloadmanager.org/ is a bit Softonic-esque, probably because of the program's heritage; the code's over at https://sourceforge.net/p/freedownload/code/HEAD/tree/trunc/, last checkin 2 weeks ago - and the project description says "we also plan to port it to Linux/OS X," which is cool)

I believe this can be achieved via a combination of httpfs [1] and archivemount [2]. Not sure about the reliability, but I think the approach is more general - you could even stream from within a remote archive this way, using whatever program you wanted. The job of an archive reader, disk reader, or "server reader" should be to translate these things into the filesystem so that all programs benefit. GNU Hurd took this approach, with the native ability to mount tarballs and FTP servers as filesystems, and POP3 as an mbox file.

Incidentally, I can't find any archives with audio hosted on a server that supports Accept-Ranges on the open web. I'd appreciate a link if anyone finds one.

Update: AVFS [3] seems to be an all-in-one solution that handles HTTP, FTP, WebDAV, Zip/RAR/Bzip, SSH, and others. Also see "podfuk" [4], though it looks pretty ancient.

[1] http://httpfs.sourceforge.net/ [2] http://www.cybernoia.de/software/archivemount/ [3] http://avf.sourceforge.net/ [4] http://atrey.karlin.mff.cuni.cz/~machek/podfuk/podfuk.html

Is Adaptive Bitrate Streaming possible?

I would have thought not.

Well, it would be possible but only if the video format in the torrent allowed for this is a way compatible with the way torrent transfers are split into blocks - i.e. it has a block structure that lines up with the torrent block structure such that block X is enough to play a given set of frames at the lowest quality, block X+1 for better quality, and so forth.

The blocks wouldn't have to be sequential: as torrent transfers are essentially random access you could have the content required for lowest quality display at the start of the file and the extra data further on, similar to how FLIF supports almost immediate low quality with progressive improvement as the rest of the file arrives (http://flif.info/ - see the video demo part way down that page, or https://uprootlabs.github.io/poly-flif/ for a more interactive look at how it copes with truncated files).

I'm not aware of any video formats that are arranged this way though. Those that are designed for adaptive streaming have a more dynamic structure that would not align well with the torrent protocols blocks.

Adaptive Bitrate Streaming works by having multiple copies of the source video at different bitrates, resolutions, and audio qualities.

Torrents are usually one file, which would not allow Adaptive Bitrate Streaming.

Pretty amazing, but burns CPU like no other.

What really sucks is that Opera added native BitTorrent support almost 10 years ago (June 2006 in Opera 9), and no other browser has had the guts since. Something to think about.

Glad you like it. It's still very beta. We're going to try to make the app more efficient. (I think it feels pretty snappy even now, though.)

You might have run into this Chrome bug. WebTorrent Desktop is an Electron app, which uses Chromium, so we inherit this bug. Chrome's WebRTC internals sometimes use 100% CPU on one core: https://bugs.chromium.org/p/chromium/issues/detail?id=392651

It will hopefully be fixed soon.

Sorry, I was referring to the in-browser version. FF 45 on Linux.

Adoption of webtorrent (bittorrent via webrtc) is almost non-existent. Unless uTorrent and libtorrent (used by Deluge, rtorrent, qbittorrent etc) will adopt this, then webtorrent in browser will never start off. Unfortunately i predict that this desktop client will not resolve this issue.

That's not true. Vuze already supports WebTorrent! Check it out: https://wiki.vuze.com/w/WebTorrent

There's also built-in support for WebTorrent (and IPFS for that matter) in the Playback video player: https://mafintosh.github.io/playback/

What i wrote is true. You know how much of market is Vuze comparing to utorrent and libtorrent?

EDIT: Here you go, data from 2015

http://lifehacker.com/5813348/five-best-bittorrent-applicati...

Azureus was something but 10 years ago. Things changed.

uTorrent shows banner ads and recently shipped with bitcoin mining malware. It's also a lot more work--and requires a second program, like VLC--to do streaming.

I'm optimistic about WebTorrent Desktop adoption.

You're right that we need lots more hybrid client adoption to bridge the BitTorrent and WebTorrent networks and really make it work in the browser.

I hope WebTorrent Desktop can be that client.

Finally, libtorrent is open source. Here's the discussion about adding WebTorrent support: https://github.com/arvidn/libtorrent/issues/223

It dosen't matter that they show banners. They are number 1 anyway. Telemetry didn't change windows adoption and ads didn't change utorrent adoption. I know about this issue from 14 october :)

I am not so optimistic, this client has 15-30 times more ram usage and couple of times more CPU usage. You can just install qBittorrent set sequential downloading and watch the movie..

Yes it matters. Ads if downloaded from Internet at start time will definitely introduce anonimity/privacy issues and cut user base by large. utorrent is #1 only because older versions (no adds/malware) are easily accessible.

I feel this is a moot point.

I'm pretty sure the install base of WebRTC data channels is far higher than BitTorrent (Chrome, Firefox, Opera and soon Edge and Safari), so existing Torrent clients providing support only helps out the existing torrents out there.

Down the line I'm willing to bet that WebTorrent grows to be more widely used than BitTorrent, just off of the lack of friction to get started for both users and developers.

You know that you can't connect to normal bittorrent clients from webtorrent in browser right? Which means that you do not have almost any sources of seeders, which means that webrtc adoption dosen't matter.

I know you can't connect to normal clients. My point is normal clients don't really matter.

"Which means that you do not have almost any sources of seeders"

All it takes for someone to seed a file is to open a web page. Think about it.

At any point if I wanted to promote a song via WebTorrent, I could build WebTorrent into an audio player and encourage my user base to click a checkbox and seed the track (toss in IndexedDB support for storing it offline and it's even better).

Complete serverless resharing with almost every benefit of BitTorrent to boot.

The only thing WebTorrent needs the existing BitTorrent install base for is the existing torrents, which don't matter as much as you might think they do (most torrents are pirated content anyways).

So yes, WebRTC adoption does matter.

If this what you wrote was true then you would see big adoption of webtorrent which is not the case. As i said before and i am saying for almost 2 years now. I love webtorrent but it NEEDS support from libtorrent and utorrent. Look at biggest bittorrent sites and on private trackers, i have seen private data on clients usage and i can tell you that adoption is almost non-existent comparing to big 5, which still means that webrtc dosen't matter in this case..

"If this what you wrote was true then you would see big adoption of webtorrent which is not the case."

You're missing the point.

Think of it this way.

Check out this video: https://fastcast.nz/videos/cityscape-chicago-ii.html

Once you're done viewing even a second of it, just know you've just adopted WebTorrent.

You don't even need to know WebTorrent exists or was used and BitTorrent wasn't needed in any way.

You know there is one peer only on server side, right? Which means they could just setup nginx and everything would be the same.

> All it takes for someone to seed a file is to open a web page. Think about it.

That's a pretty big hurdle considering most regular torrent users don't even realize WebTorrent exists, and these users contribute the vast majority of the content available on BitTorrent.

Of course in the ideal world where WebTorrent has significant market share, this wouldn't be nearly as much of a problem. But I agree with lossolo that without interop with libtorrent at the very least (if not uTorrent as well), that ideal world is probably never going to come into existence.

"That's a pretty big hurdle considering most regular torrent users don't even realize WebTorrent exists"

They don't need to. That's the point.

If I send a user to https://file.pizza/ to transfer a file to them. Not once to I have to mention that it uses WebTorrent. IMO that's the killer feature.

Yes, WebTorrent enables plenty of really great, novel use cases for BitTorrent technology, but it will remain a second-rate BitTorrent client in the eyes of most existing BitTorrent users without great interop. Although I personally do like these new WebTorrent-based web services, there's no guarantee that any of them will ever reach critical mass in terms of adoption, whereas BitTorrent itself has long since reached critical mass. To say that "normal clients don't really matter" simply because WebTorrent can enable new use cases for BitTorrent (that haven't shown any significant traction) really seems like you're missing the proverbial forest for the trees.

> To say that "normal clients don't really matter" simply because WebTorrent can enable new use cases for BitTorrent (that haven't shown any significant traction) really seems like you're missing the proverbial forest for the trees.

I already said "the forest" doesn't matter.

Seriously, if I were building something with WebTorrent. Let's say Netflix (Note: https://torrentfreak.com/webtorrent-brings-bittorrent-to-the...). And most of the existing BitTorrent seeds are pirated content.

Why would I care?

"there's no guarantee that any of them will ever reach critical mass in terms of adoption"

When you give away tool with a widespread guaranteed userbase like this, someone is bound to build something that people will use. Not to say WebTorrent will be this big, but that's how the internet came to be.

And this is a completely decentralized upload and download library for the web, someone's going to build something popular with it I'm sure and it's not even at version 1.0 yet.

Seems like a no-brainer for TPB/etc to adopt WebTorrent. That should get the momentum going.

Agreed, moot point. All it would take are a few large companies to hoist its downloads through this platform and provide the initial seed pool...

First candidate: Blizzard!

I looked into the complexity of implementing webtorrent (which is just websocket signaling and webrtc data channel connections to peers rather than the traditional http signaling with udp to peers) in libtorrent and it is kind of a giant mess of C++ :p

The problem is more that every desktop client, for some reason, has also reimplemented bittorrent. Specifically, Transmission, Deluge, and KTorrent all use their own bittorrent protocol implementations, and none of them are particularly or easily extensible enough to provide webtorrent support easily.

But let it be known eyes are on the prize and have at least looked into it, it is just a hard problem, exacerbated by implementation fragmentation.

Take note developers of similar things - the use of freely licensed content in your demo screenshot might save your ass legally.

This is something many people seem to screw up - just yesterday a similar torrent streaming tool had pirated screenshots. The OpenBazaar guys actually suggested using it for illegal things at one point... Like it or not, courts have recognized a difference between soneone who built a tool for legal uses and someone who built a tool for piracy.

Since Popcorn Time I've been confronted with a rising number of people who have no idea that those "streaming" portals are also uploading. I'm being confronted because they got a letter from a friendly lawyer telling them to pay a lot of money.

I hope you are communicating that fact properly.

how does the webtorrent team respond to criticisms of sequential downloading in torrents?

while i personally believe folks should mirror & seed torrents regardless of how others consume these torrents, i'm curious about the developers' perspectives on this issue

in any case, this desktop app looks very slick, thanks webtorrent team for all work pushing webrtc and torrenting forward

* It only download sequentially when you're streaming. If you're just downloading a torrent, it will use rarest-first.

* We plan to switch to a hybrid strategy: it will download the blocks it needs, but once enough video is buffered, it will transition to rarest-first.

* It doesn't really matter. This used to be more of an issue back when torrents took hours or days to download and upload speeds were slow. If you download the entire torrent in five minutes and then seed it for days, it doesn't matter for the health of the swarm what order you downloaded the pieces in.

This might be more complexity than you are ready for, but here's what I suggest:

* Remove the notion of two different modes. * Remove the ability to switch between two modes. Because with this capability, people will inevitably switch it to the greedy mode, even if they have no need to, out of an inane fear that they might need to. * Implement your hybrid mode, again without a "switching" mechanism: * In your implementation of "rarest first", artificially apply extra weight to the next N segments that will be needed for the current play point.

I think this will result in very good behavior, in all cases. It is completely hands-off. Users tight on bandwidth will inevitably spend all of their bandwidth downloading the extra weighted segments needed to place soon. Users with any spare bandwidth will easily get past the next and segments, and then the rarest segments will automatically be the next fetched. All without any switches for users to set wrong.

This is an excellent suggestion – we're actually planning to do basically what you describe.

And you're totally right that the user shouldn't need to switch between the modes manually. There's no UI for that option, and I don't think there should be. That's just an option of the underlying WebTorrent torrent engine (https://github.com/feross/webtorrent), though even that should probably be removed. :)

WebTorrent supports two download modes: sequential and rarest-first.

In sequential mode, pieces are fetched with high priority when they're immediately needed for streaming. In rarest-first mode, the traditional approach is used.

We're going to add automatic mode switching to a future release, so rarest-first will be used most of the time. The only time sequential mode is needed is right when streaming begins and every time a seek happens. I think this is a nice balance between helping the local user and keeping the swarm healthy.

Progressive web apps (which work offline, btw) can already be installed on Windows (demo video: https://www.youtube.com/watch?v=VUma1l3BKOI) via Chrome — all the app needs is a manifest and a service worker — so it might be worth experimenting with supporting it (for people who’ve “grown out of” installers).

These native apps powered by electron allow for much better desktop integration than the kind of web application you're talking about here. Traditional installers are required for these kind of desktop integration that most people have grown accustomed to. It's a trade-off!

That and they wouldn't get classic BitTorrent compatibility in a web app.

I thought WebTorrent already is BitTorrent-compatible, no?

No, WebRTC cannot make normal TCP connections so it can't be used to connect to a classic client. In theory all bittorrent software could implement WebRTC connections though

What about Peerflix (https://github.com/mafintosh/peerflix)? This has been around for a while. You can actually play your videos/other stuff in the media player of your likings, like VLC, which is a mature and stable video player. Why build an entire video player so you can stream..

feels pretty and nice. really easy to share files with this too (no more waiting to upload to a host, etc etc)

Once this gets the standard set of features (file selection being the main one for me, since datacaps exist here, and spending 14 GB to watch the first ep and then decide I don't like whatever I'm watching is a bit of a pain) (also compact mode), I think this might very-well become my go-to client

There's a bunch of these already but none of them allow you to limit bandwidth so they are unusable on residential internet in most cases. Baffles me...

This is a planned feature. Will be fixed soon: https://github.com/feross/webtorrent/issues/163

WebTorrent CLI: https://github.com/feross/webtorrent-cli WebTorrent Hybrid: https://github.com/feross/webtorrent-hybrid

Very cool announce!

Do you think running webtorrent under Duktape could work (with appropriate network APIs) ?

Perhaps. You'd want to use the npm package directly for that. https://github.com/feross/webtorrent

If there's any changes you need to make it work for, just open an issue or send a PR!

Just gave this a spin and really loved the ease of use, but I can't quite replace my go-to torrent client (currently Hadouken) with it just yet. Here are some suggestions for future improvements to consider in case they're not on your shortlist already:

1) Ability to monitor multiple folders and automatically add torrents (and ideally the ability to specify a different download location for each)

2) Ability to run as a headless service/daemon with a thin management UI like Deluge/Transmission/Hadouken/etc

> Ability to monitor multiple folders and automatically add torrents

I don't think this is a bad idea, but that's definitely a feature for v2.

2) Ability to run as a headless service/daemon with a thin management UI like Deluge/Transmission/Hadouken/etc reply

This isn't exactly what you're asking for, but you might appreciate webtorrent-cli: https://github.com/feross/webtorrent-cli which works entirely from the command line.

Excellent work; gorgeous website! Congratulations on the release.

Will browsers prompt (like mic and video) for permission before opening p2p traffic? I'm afraid random sites could behind my back make my machine (via nat traversal if needed) participate in illegal torrenting. It's the legal world we live in, and while I welcome the new browser abstractions, this must be behind a permission popup and provide a way to permanently disable without building a custom browser. What's the status on that?

I think opensubtitles integration can be a good feature too.

Probably the best WebRTC app made with Electron! Congrats to Feross & DC!

Guys, would you like to present at the next SF WebRTC Meetup on April 28th? http://www.meetup.com/SF-WebRTC/events/229430038/

Yes, we'll do it :) Email me!

I still prefer peerflix over stuff like this. Small CLI app and you can use a superior video player (VLC, SMPLayer, mpv etc).

I just tried it out. It's pretty cool but for some reason there was no audio. Its going to be interesting to see how to guys handle the hodgepodge of video and audio codecs out there.

Unless I am mistaken, this does not appear to have a way to easily limit upload/download rate, which would be a deal breaker for me.

Happy to be proven wrong on this of course.

+1 Would definitely use if I would be able to cut the upload entirely. Currently under jurisdiction that prohibits sharing illegal content, while perfectly legal to download any music/videos for personal use only if available in the network.

Lacks a usable volume control, and the one in the View options seems non-linear.

Also showed a rather improbable download speed for my home line.

> Lacks a usable volume control

There's a pull request for that here: https://github.com/feross/webtorrent-desktop/pull/330 It will be in the next release.

> Showed a rather improbable download speed for my home line.

Thanks for the compliment. We try hard ;)

Do torrent trackers need support for WebTorrent before they work with WebTorrent clients?

Keep up the great fucking work!!!

The website is so beautiful, I want to bookmark it.

Damn this is cool.

Not fucking again! This is not a torrent client. It cannot speak to any other existing client. The browser is not the right place for a torrent client.

Incorrect. This client knows both the WebTorrent and BitTorrent protocols, bridging the two networks.

Reply on news.ycombinator.com