Show HN: WebRTC implementation for Python using Asyncio | Hacker News Reader

Show HN: WebRTC implementation for Python using Asyncio

github.com

216 pointsjlaine51 comments

OMG this is a godsend! I've wanted to do webrtc outside of a browser in python for a while now but there was no available implementations (besides the native one, and I didn't want to go down the swig route).

Some questions:

Can it handle the signalling on its own?

Can it substitute any part of a webrtc session? That is, can I have either browser-aiortc, aiortc-browser or aiortc-aiortc?

Can it be used to build, say, a desktop application doing webrtc communication?

Signaling is a very app-specific question, as you can use a wide range of options. If you take a look at the "apprtc" example, it's a demo which uses appr.tc's signaling.

You can indeed use aiortc facing a browser, itself or any valid WebRTC endpoint.

Sure, you can use aiortc for a desktop application. However for this specific usecase embedding a browser which supports WebRTC (say QtWebEngine) might be a good option too.

Naive question: what's cool about doing webrtc outside a browser?

If you are trying to build e.g. Twitch or Discord, you want to have a server backend to handle processing, hide user IPs from each other for privacy etc. (the term you are looking for is single forwarding unit). If you just want to get something up quickly, I recommend OpenVidu.

If you're hiding everything behind a server, couldn't you just use WebSockets? I thought the whole appeal of WebRTC is the whole peer to peer setup.

WebRTC provides audio and video streaming, as well as unreliable/unordered data transport through data channels, which is really great for games.

You can't use UDP with web sockets.

Maybe you want to do something audio/video based, don't want to reinvent the wheel and more importantly you absolutely don't want a web browser running on the system running your application (for example because you're running on an embedded system)

Nice work! As a WebRTC implementer (both native and in Chrome), I applaud the work!

But I can only notice quite a bit of mismatch between your APIs and the current spec. Do you plan on updating your implementation to match it?

Is there any specific area you spotted which would need some love? Modern spec compliance is most definitely a goal (e.g. RTCRtpTransceiver has been there from the start, aiortc uses the "modern" SDP form for data-channels), so any pointers are greatly appreciated!

I've had some great interactions with the Mozilla crowd (hey Lennart, hey Nils) - looking forward to more of the same!

Anything related to RTCRtpSendParameters (previously RTCRtpEncodingParameters) really.

For example, RTCRtpSender.getParameters is missing and the setParameters seems to be missing (unless that's the send function). And as far as I can tell, you are not supposed to call setParameters without getParameters first, so that's not spec compliant ;)

And of course, the sendEncodings property of the init parameters in addTransceiver is missing. It is required to setup simulcast (and later SVC when we standardize it).

Duly noted, I've opened an issue to track this:

https://github.com/jlaine/aiortc/issues/76

aiortc's RTCPeerConnection indeed uses RTCRtpSender.send to apply parameters. This is directly inspired by ORTC:

http://draft.ortc.org/#dom-rtcrtpsender-send

Also, my take on this is that you should probably not mix WebRTC and ORTC APIs. Those are quite different (even if there are similarities).

If you want to implement ORTC, fine, but you should probably do it in another package and make sure you don't expose fields in WebRTC that aren't part of the standard. Some fields (like the SSRC field) is not exposed for good reasons.

This is super cool, great work! It's the second (mostly-)language-native implementation[1] outside of a browser that I'm aware of, both started this year despite the WebRTC spec being finalized about 5 years ago.

It's really important for libraries like this to exist so more people can build apps without binding to massive hard-to-compile browser codebases. Thank you for working on it!

Also kudos for going asyncio and Python 3 right from the gate.

[1] At least semi-functioning with DataChannel support. Lots of started attempts that have never gotten this far. The other implementation is in Go: https://github.com/pions/webrtc

Since a lot of WebRTC devs may be in this thread (and I personally didn't know of this project nor https://github.com/pions/webrtc) -- any erlang/elixir implementations out there?

The members of Slack‘s Calls team have written their own and they have spoken at SF HTML5 meet-up but nothing else publicised AFAIK.

Also somebody has forked erlyvideo to deal with WebRTC

Wow! This is hard to do! I got most of an ICE implementation done in Node.js then got completely hung up on DTLS. Kudos!

Hi can the author or someone point to a hello world code using end-to-end of the above code?

There are several examples available from the project's github repo:

https://github.com/jlaine/aiortc/tree/master/examples

I personally like the "server" and "apprtc" demos as you can easily talk to a browser. They illustrate slightly different things:

- "apprtc" relies on a third-party signaling service, and shows you how you can play media from a file / record it to a file, or generate video frame-by-frame.

- "server" shows you how you can combine media and signaling into a single Python-based server, and how you can apply image processing on the fly to the received video.

Great work! I can really appreciate the effort you likely put into re-implementing SCTP, since I'm currently working on implementing SCTP in Rust for the same reasons (WebRTC support). :)

That's good news, the WebRTC world is too much of a libusrsctp monoculture to my taste at the moment. Don't be surprised if you find some interesting edge cases in browsers as you stray from the beaten track :)

https://bugzilla.mozilla.org/show_bug.cgi?id=1443032

Yipes! That's good to know about, thanks. At the same time, I've also been surprised at how lenient the browsers are at accepting associations from my incomplete implementation (e.g. without the required SCTP extensions, etc.).

I can confirm you can get data channels to work with zero SCTP extensions, I only added them afterwards!

One implementation (libusrsctp-based) you might want to interop with is rawrtc. Like aiortc's datachannel-cli example, it features copy-and-paste signaling which makes testing easy.

https://github.com/rawrtc/rawrtc

Lennart, rawrtc's author is very knowledgeable in all things DataChannels and active in the standardization process.

I'm really eager for a SCTP/WebRTC implementation in Rust! I took a stab at it, and re-implementing SCTP was the obstacle that stalled me.

Is your work on github/gitlab?

Yes [1], but it's incomplete, needs a lot of TODO resolution and cleanup, (insert other scary disclaimers here), etc. I also have some slides from a presentation [2].

[1] https://github.com/simmons/webrtc-sctp [2] https://cafbit.com/post/rust_webrtc_data_channels/

If the reason you guys want a WebRTC implementation in Rust is to be able to write WebRTC server software in Rust, you might be interested in Janus -- it's a plugin-based C WebRTC server with a C ABI for plugins that you can easily write in Rust.

That's awesome simmons! I can't wait to see Rust/Python/Go/Node all talking directly to each other. I really want to see a world where P2P communication is common.

It could really change how people design systems, have servers exchange data directly instead of overloading message brokers. Could have an even bigger impact on users! Instead we could see users exchanging files directly instead of depending on paid services!

I am the author of https://github.com/pions/webrtc we implemented SCTP, SRTP and working on DTLS now. So if you ever want to grab anything please do :) and always happy to chat if there is anything that could be better/any way we could help the rust implementation

> working on DTLS now

Any info or branch where I can watch? I am watching your project closely and curious about the OpenSSL requirement for DTLS considering Go's quality crypto stack. Is the work you're talking about for removing this dependency? If not, I might take a stab at it.

https://github.com/sean-der/dtls

There isn't a DTLS implementation right now that is maintained, and adopting one would be more work then starting from scratch. I didn't know anything about TLS before I started, so learning as I go.

I would love to work together on this! If you want to open a PR/issue I can give you commit access. Also join the Gophers slack and join the #pion channel, there is plenty of work to do and would love to split it with someone else :)

Yes this will remove the OpenSSL dependency! I am really excited when people can build for any platform that Go is supported on. After that I also want to start shipping C APIs, hoping people can build lots of cool things with it then.

I'll take a peek, but my schedule may not give me much time at the moment.

> After that I also want to start shipping C APIs, hoping people can build lots of cool things with it then.

I've found Go's FFI export story to be weak, especially on Windows.

> learning as I go

I am as well. After having read the spec and seeing its differences from TCP-based TLS, I may have a go, but not sure when. I'll give a shout if I get anywhere of note.

This is really impressive! Seems like a great way to learn WebRTC and potentially is going to be much more portable than the reference C++ implementation.

Isn’t C++ generally more portable than python? All you need is a sufficient compiler, as opposed to a correctly set up Python installation of the correct version.

Well, CPython is C89, that level of compatibility is hard to beat.

I suppose many readers of HN already know this, but CPython is the name of the original and most popular implementation of Python; it is the implementation found at python.org. It is written in C and hence is called CPython.

Sorry, what I meant in this case by portable was being able to translate this code to another platform. The C++ implementation of WebRTC is incredibly complex.

Great work! Thanks for this. Scripting WebRTC is always a hassle. And the CV integration will make things easier ;)

Absolutely, might as well enjoy the rich Python ecosystem!

Starting with aiortc 0.9.9, there is now also deep integration with PyAV for all your FFmpeg needs.

- At the lowest level : aiortc uses PyAV's AudioFrame and VideoFrame class, and PyAV is soon to gain more ndarray converters: https://github.com/mikeboers/PyAV/pull/415

- At a higher level : aiortc provides MediaPlayer and MediaRecorder classes to read or write audio/video

Amazing stuff! Since I mainly do python, I'll definitely dig into this over the weekend.

Can this library in the current state record audio of both local and remote streams at the same time and could you mute a user for every participant on the call?

Can you comment on any performance hotspots due to Python (are there any)? What wouldn't this library be suitable for?

So far the main CPU hog is (unsurprisingly) media encoding, especially video. Luckily this occurs off the main thread, inside C function calls. Unless I'm mistaken the GIL is released, so there is no Python-specific downside on this specific point.

I've found heavy usage of asyncio.Queue to bridge consumers / producers does not lead to amazing performance, so I'm trying to cut down on that pattern in favour of.. callbacks (I know - not very asyncio-ish). A good testcase for testing performance without media encoding is transferring large amounts of data over datachannels (see datachannel-filexfer example). I usually hit ~ 70Mbps on the loopback interface, which is more than I've managed to get out of browsers.

I can't say I have experience with using aiortc at scale yet, so there are almost certainly so hotspots I haven't seen.

Would have thought that using kernel SCTP would be much faster.

WebRTC afaik uses SCTP wrapped in UDP, not SCTP directly.

Doesn't have to wrap it in UDP. Seems to be just an expectation that firewalls can't handle SCTP directly so you will need to wrap it in something else. Extending a firewall to understand SCTP isn't particularly difficult.

In any case, the SCTP reference implementation in FreeBSD can do RFC 6951 encapsulation, someone could add it to Linux too.

It's specified as wrapping in DTLS/UDP, so implementations have to do that if they want to be interopable. The "just an expectation" is sadly a well-studied fact.

That FreeBSD can do that is neat, but just it doing that is pretty much the answer why a library wouldn't implement it, or at least not as a priority.

You then need to try and connect via SCTP and then fallback to SCTP-over-UDP (given Windows/macOS don't support SCTP and you need SYSTEM/root access to create a raw socket to implement it in userland, which is a non-starter), which has a performance cost in a common case (falling back).

Actually that would be SCTP over DTLS over UDP (see RTCDtlsTransport class), and a QUIC transport might be an option in the future:

https://groups.google.com/a/chromium.org/forum/#!topic/blink...

This is super cool. For what are you using it?

Is ORTC used by something else than Skype?

ORTC is what Microsoft has chosen to implement in Edge, so if you use WebRTC in Edge, you're most likely using the WebRTC-over-ORTC adaptor.

In short ORTC is: break WebRTC into objects you manipulate directly, instead of everything being instanciated via RTCPeerConnection. There is significant overlap between the two specs, for instance RTCRtpReceiver / RTCRtpSender.

Awesome work!

Reply on news.ycombinator.com