FFmpeg merges WebRTC support
git.ffmpeg.org
git.ffmpeg.org
Now that GStreamer, OBS and FFmpeg all have WHIP support we finally have a ubiquitous protocol for video broadcasting for all platforms (Mobile, Web, Embedded, Broadcasting Software etc...)
I have been working on Open Source + WebRTC Broadcasting for years now. This is a huge milestone :)
[0] https://github.com/Glimesh/broadcast-box?tab=readme-ov-file#...
When are you coming back to the WebRTC space, lots more cool stuff you could b doing :) I really loved [0] it's so cool that a user can access a server behind a firewall/NAT without setting up a VPN or having SSH constantly listening.
Plex and ffmpeg, perfectly fine. VLC is not a fan.
gst-launch-1.0 playbin3 uri="gstwebrtc://localhost:8443?peer-id=<webrtcsink-peer-id>"
[1] https://gstreamer.freedesktop.org/documentation/rswebrtc/web...
I've been struggling to get a proper low-latency screen+audio recording going (on macos) and streaming that over WebRTC. Either the audio gets de-sync, or the streaming latency is too high.
but this+obs+a webapp for input+ydotool to pass the input to dosbox. then i can just open a page on the browser on the phone.
Are we entering an era where you don't need Amazon's budget to host something like Twitch?
Before you needed to run expensive transcoding jobs to be able to support heterogenous clients. Once we get Simulcast the only cost will be bandwidth.
With Hetzner I am paying $1 a TB. With AV1 or H265 + Simulcast I am getting 4K for hundreds of users on just a single server.
We will have some growing pains, but I am not giving up until I can make this accessible to everyone.
So if you've got ~18 Gbps of upload bandwidth you're ready for 10,000-20,000 viewers.
Assuming a single gigabit symmetric connection could dedicate at most 100mb of upload bandwidth, you'd need one such viewer per each 25 viewers with a worse connection. This feels achievable.
You'd have 1 server that broadcasts to at most 10K tier-1 viewers. Tier 2 viewers get 3 tier1 ips for failover, and pre-negotiate a connection with them, e.g. through STUn, to get sub-second failovers in case their primary source quits the stream. A central control plane load balances these.
With something like 15s of buffer (acceptable for a gaming stream, not so much for sports, where your neighbors might be watching on satellite and cheer), this feels achievable.
Are all of the issues P2P brings really worth it?
I'd say this definitely opens up streaming from your desktop with 1 CPU core handling a SFU for say 100 viewers = 500Mb or from a $5/month VPS if you've not got the capacity at home. That's pretty awesome, for most people no need to use P2P.
CGNAT is going to make that a hassle.
i'm so effing tired of NAT here and NAT there and NATs in between NATs. NAT complicates things for almost no reason now that ipv6 was released almost 30 years ago. Wait, what? 30 years? Now I'm VERY shocked at how little ipv6 has been deployed.
You don't even need gigabit connections to make p2p work: a video stream is usually between 1-10mbps, with which you can do p2p even with a regular connection.
(I used to work for a start-up doing p2p streaming using WebRTC in the browser, we got up to 90% efficiency on live streaming back in 2015, and 80% for VoD).
I'm still very confused that this technology hasn't become mainstream (there are privacy concerns, but that doesn't usually stop the tech market…).
If we assumed an average of 6Mb per stream that's 3000 streams, practically speaking a little lower.
It's all relative I guess but it's not that high.
https://networkengineering.stackexchange.com/questions/47994...
Now I see this news?! Perfect timing. (:
I like this guy.
ICE (protocol for networking) supports this today. It just needs to get into the software.
I hope some day we can switch to a QUIC or WebTransport based p2p protocol, rather than use SCTP. QUIC does the SCTP job very well atop existing UDP, rather than add such wild complexity & variance. One candidate, Media-over-Quic ?MoQ), but the browser doesn't have a p2p quic & progress on that stalled out years ago. https://quic.video/ https://datatracker.ietf.org/group/moq/about/
Most 'WHIP Providers' also support DataChannel. But it isn't a standardized thing yet
Hypothetically ffmpeg could be an ICE server for peer-connecting, do SDP for stream negotiation possibly with a side of WHEP (egress protocol) as well, could do SCTP for actual stream transfer. Such that it could sort of act as a standalone peer, rather than offload that work to a gateway service.
Worth noting that gstreamer & OBS also are WHIP based, rely on an external gateway for their WebRTC support. There's not one clear way to do a bunch of the WebRTC layer cake (albeit WHEP is fairly popular I think at this point?), so WHIP is a good way to support sending videos, without having to make a bunch of other decisions that may or may not jive with how someone wants to implement WebRTC in their system; those decisions are all in the WHIP gateway. It may be better to decouple, not try to do it all, which would require specific opinionative approaches.
Phoronix has a somewhat more informative page: https://www.phoronix.com/news/FFmpeg-Lands-WHIP-Muxer
WHIP is 'pushing media via WebRTC' - https://www.ietf.org/archive/id/draft-ietf-wish-whip-01.html
WHEP is 'pulling media via WebRTC' - https://github.com/wish-wg/webrtc-http-egress-protocol/blob/...
WHEP isn't standardized/still changing a bit. After it lands I will try and get it into OBS/FFmpeg (and more)
ffmpeg also has some processing abilities of its own, so you could e.g. greenscreen (chroma key) from a stream onto an existing video background.
ffmpeg is a pretty low-level building block and as others have said, it's mostly used as a library - a lot of video players or processing tools can now add support for stream inputs easily, and that's probably where the biggest impact is.
Say you wanted to do a virtual portal installation connecting views from two different cities with live audio, you could have ffmpeg feed off a professional cinema or DSLR camera device with a clean audio feed and stream that over WebRTC into a webpage-based live viewer.
Or say you wanna do a webpage that remote controls a drone or rover robot, it would be great for that.
(I did sub-100ms glass-to-glass streaming with VLC back in the day, so it is eminently possible. But the browser is in your way.)
[1] Much less than that is going to be tricky under non-perfect network conditions, because once you start having any sort of packet drop, you want to go from TCP's retransmission regime and instead start dropping packets, take the artifacts for a little while, and then go on.
node1# nc -u node2 12345 < /dev/fb0
node2# nc -lu 12345 > /dev/fb0
We could go make a better/simpler standard for video streaming that is TCP. What a giant battle though that would never seen adoption .
I have accepted/embraced the challenge of making WebRTC as accessible as possible. Stuff like WebRTC for the Curious in hopes of making it less painful for everyone dealing with the complexity :)
What do you mean? <video> in HTTP against a stream works, you don't need a new standard. But it's not a low-latency path (you cannot control the buffer).
Apple’s LL-HLS spec shrinks those segments into “partial segments” and uses CMAF to let the player start decoding while a chunk is still arriving. With careful encoder tuning, HTTP/2/3 push, CDN support, and a compatible player you can reach 2–5 s, sometimes a bit lower—but every link in the chain has to cooperate, so implementations are still called “heroic” for a reason.
Safari plays HLS natively and on other browsers you can bolt on hls.js via Media Source Extensions. DASH, RTMP, SRT, etc. all need extra code or plugins, so HLS became the “safe default” for on-demand and broadcast-style streaming even though it isn’t low-latency friendly.
webrtcHacks has an article on it[0] kind of old, but captures the spirit of it!
If you know how to use it ffmpeg is such an amazing stand alone/plug and play piece of media software.
Especially with Simulcast it will make it SO cheap/easy for people.
I made https://github.com/Glimesh/broadcast-box in a hope to make self-hosting + WebRTC a lot easier :)
Yesterday I asked it for a command to take a 14 minute video, play the first 10 seconds in realtime, and rest at 10x speed. The ffmpeg CLI syntax always seemed to be able to do anything if you could keep it all in you head, but I was still surprised to see that ffmpeg could do it all in one command.
Makes life much more easier when I can play video games or read books without having to check status every 20 mins.
Though I haven't created as many as you have. Would you mind sharing some of them??
One that I find regular use for when copy+paste does not work because I am either connected to a terminal emulation, vm, or something of the like is typeitforme. It takes the contents of a text file and sends them through the keyboard buffer after a few second delay (that allows me time to switch focus to the window I want the typing done).
I currently have it as a entry in my ~/.bash_aliasas file.
alias typeitforme='sleep 3 && xdotool type --file '
This works in stock Ubuntu Linux. You can check out the xdotool documentation for ideas how to refine it to do more.
That's LLMs for you
Now it is all wallet garden/app-per-service.
This implementation is very small. I feel 100% confident we are giving users the best thing possible.
Of course, you're right that this implementation is very small. It's very different than a typical client implementation, I don't share the same concerns. It's also only the WHIP portion of WebRTC, and anyone processing user input through ffmpeg is hopefully compiling a version enabling only the features they use, or at least "--disable-muxer=whip" and others at configure time. Or, you know, you could specify everything explicitly at runtime so ffmpeg won't load features based on variable user input.
Like what? I did a quick search and most seem to be stuff like ip leaks and fingerprinting, which isn't relevant in ffmpeg.
While WebRTC causes fingerprinting risks in browsers, isn’t that unrelated to running ffmpeg?
* CVE-2015-1260
* CVE-2022-4924
* CVE-2023-7010
* CVE-2023-7024
* CVE-2024-3170
* CVE-2024-4764
* CVE-2024-5493
* CVE-2024-10488
Of course, I agree that it's not relevant to ffmpeg. But seeing "WebRTC" triggers the same part of the brain that looks out for unescaped SQL statements. Good opportunity to point out the difference in this implementation.
This is like searching CVE for “node” and then claiming Node is terrible because some node packages have vulnerabilities. Low effort and intended to fit evidence to an opinion instead of evaluating evidence. “Linux” has 17,000 results; using your critical lens, all Linux is insecure.
We're discussing whether it's right to take a second look from a security standpoint when a software implements WebRTC. In this case, it's nuanced, and the implementation in FFmpeg is very different than the more complete implementations you find in browsers. And when browsers have implemented WebRTC, many vulnerabilities have followed.
So the double-take is justified here, even if only in principle. No one is saying WebRTC is insecure, or FFmpeg, or node, or Linux..........
I did a cursory read of each CVE. Wherever you got the idea I did not, you must have forgot to include it in your post. Just now, I picked one from random. It reports "Multiple WebRTC threads could have claimed a newly connected audio input leading to use-after-free."
Does that exactly qualify as an "implementation bug"? I don't know, and I don't care, because how you taxonomize a CVE has nothing to do with whether it's a vulnerability that was introduced when implementing WebRTC. And it is.
“No one is saying webrtc is insecure”? That is literally what the comment was doing, which you attempted to legitimize by listing browser-specific CVEs.
Someone pointed to a car fire and said gasoline caused the fire, and you posted pictures of car fires. There is a reason a Fire Investigator (like a security researcher would) considers the difference between what started a fire and an accellerant. WebRTC was not the cause of these vulnerabilities like you are trying to imply and like the opinion you attempted to legitimize.
“I don’t care” — clearly, if you couldn’t take the time to understand the difference, I’m not surprised.
If ffmpeg implements WHEP in the future then I'd certainly be concerned about both of those things when viewing a stream. Probably less so for serving a stream up, particularly via a gateway (the current implementation IIUC).
You're right that biggest reason people usually recommend disabling it is to prevent your IP from leaking when using a VPN https://www.techradar.com/vpn/webrtc-leaks but not having to worry about RCE or DoS is a nice bonus
I'm not sure how much will this impact ffmpeg users. Considering that WebRTC has a bad track record in terms of security though, I do worry a little that its inclusion in one more place on our systems could increase attack surface.
That would be like saying saying 'webrtc is more secure then http' by posting this https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=http
IP Leaking has been fixed since 2019[0]. ICE/P2P is still a huge attack surface though. I have seen lots of other tricks being tried.
ffmpeg has had so many issues in the past [1], it's best practice anyway to keep it well contained when dealing with user input. Create a docker image with nothing but ffmpeg and its dependencies installed and do a "docker run" for every transcode job you got. Or maybe add ClamAV, OpenOffice and ImageMagick in the image as well if you also need to deal with creating thumbnails of images and document.
And personally, I'd go a step further and keep the servers that deal with user-generated files in more than accepting and serving them in their own, heavily locked down VLAN (or Security Group if you're on AWS).
That's not a dumbass criticism of any of these projects mentioned by the way. Security is hard, especially when dealing with binary formats that have inherited a lot of sometimes questionably reverse engineered garbage. It's wise to recognize this before getting fucked over like 4chan was.
To be more serious seurity is often in conflict with simplicity, efficiency, usability, and many other good things.
A baseline level of security (and avoidance of insecurities) should be expected everywhere, docker allows many places to easily reach it and is often a good enough tradeoff for many realities.
but I agree.
Full transcoding would be a bit more complex, but assuming decoding is done in software, I think that should also be possible.
Anubis let me go
edit: Trying again a few minutes later worked
Perhaps "common and technically works" would be a better way to put that (similarly for rebase). I suspect people would stop squashing if git gained the ability to tag groups of commits with topics in either a nested or overlapping manner.
You are being downvoted, but you are entirely correct. This is also explicitly not allowed in FFmpeg, but this was pushed after many months, with no heads up on the list, no final review sign off, and with some developers expressing (and continuing to express) reservations about its quality on the list and IRC.
The biggest thing missing right now is NACK support, and one of the authors has said they intend to do this (along with fixing old OpenSSL version support, and supporting other libraries). Until that is done, it isn't really "prod ready", so to speak.
For some context, there has been a history of half-supported things being pushed to FFmpeg by companies or people who just need some subset of $thing, in the past, and vendors using that to sell their products with "FFmpeg isn't good enough" marketing, while the feature is either brought up to standard, or in some cases, removed, as the original authors vanish, so it's perhaps a touchy subject for us :) (and why my post was perhaps unnecessarily grumpy).
As for the git / premature push stuff, I strongly believe it is a knock-on effect of mailing list based development - the team working on this support did it elsewhere, and had a designated person send it to the list, meaning every bit of communication is garbled. But that is a whole different can of worms :D.
Quick! Quick! I need to find something bad about it... wait... AH!
Does it compile with the latest libressl? Hopefully not (like python _ssl.c) and I can start talking bad about it.
;P
Ofc, that was irony.
We all know the main issue with webRTC is not its implementations, but webRTC itlself.
All that said, it is exactly at this very time twitch.tv chose to break ffmpeg HLS (its current beta HLS streams are completely breaking ffmpeg HLS support...).
I'm still waiting for ffmpeg CLI tool to merge pipewire + xdg-desktop-portal support. You still can't record a screen or window on Wayland with it.
With bitwhip[0] I got it way lower then that even.
https://github.com/pion/webrtc/tree/master/examples/save-to-... here is an example of a server that does what you need on the backend
Nope. Get lost. Running random code from websites you don't know is asking for desaster.
Reddit lost their own community's trust when the CEO ejected the community's moderators.
Information posted there is now far less likely to be qualitative compared to other places, so what's the point of going there?
You could also join the Pion one https://pion.ly/discord
Other place is video-dev Slack https://www.video-dev.org/
X also has a great community of hackers!
Too bad nobody is using open forums (e.g. Discourse, Matrix, Mastodon).
I do it myself too. I boycotted StackOverflow for the past 13 years, never sharing another solution with anyone because the admins on that site were hounding me about posting CORRECT answers without first checking to see if a similar answer already existed. I vowed never to help anyone on there ever again, after that. I told them, even back then, that the more correct info they collect for free, the better, because some day AI would sifting thru it, and that they were being stupid. As of 2025, I was finally proven right.
Make a simple example of speeding up an mp4
video in the browser using a version of ffmpeg
that runs in the browser. Don't use any server
side tech like node. Make it a single html file.
I'm convinced that programmers' bias towards LLMs is strongly correlated with the weirdness of their work. Very often my strange ideas pushed to LLMs look like solutions but are rather broken and hallucinated attempts which only vaguely represent what needs to be done.
This is an extremely astute observation; my work has always been somewhat weird and I've never found LLMs to be more then an interesting party-trick
https://ffmpegwasm.netlify.app/docs/getting-started/usage
It transcodes a webm file to MP4, but making it speed up the video is trivial: just add arguments to `ffmpeg.exec()`. Your lack of success in this task is trusting an LLM to know about cutting-edge libraries and how to use them, not a lack of progress in the area.
You say "an environment with React". My environment is the browser.
I don't know how one is supposed to run that nameless function on that page. What I am looking for is a simple, complete example in HTML that can run standalone when opened in the browser. Without any server side processing involved.
It might have to do with these two strange comments at the top:
// import { FFmpeg } from '@ffmpeg/ffmpeg';
// import { fetchFile, toBlobURL } from '@ffmpeg/util';
By default, the build relies on web workers which need to load their code from somewhere (and usually it has to be the same origin as the code making the request)
Through much mastery of JS build systems that I would not wish on my enemies, I bet you could get it working on localhost, but you’ll have a much better time of it if you set up vite or something for a local build. You can still easily do a “serverless” deploy with GitHub pages or similar but you do need an http server correctly configured for asset requests.
https://ffmpegwasm.netlify.app/docs/getting-started/usage/
If you can't be arsed to google the library and read the Usage page and run the _one command_ on the Installation page to come up with a working example (or: tweak the single line of the sample code in the live editor in the docs to do what you want it to do), how do you expect to do anything beyond "an easy PoC"? At what point does your inability/unwillingness to do single-digit-minutes of effort to explore an idea really just mean you aren't the right person for the job? Hell, even just pasting the code sample into the LLM and asking it to change it for you would get you to the right answer.
I don't think so as I did not get it to run. And if he really can accomplish it with copy+paste, why wouldn't he demonstrate it?
"Tap with a hammer: $1. Knowing where to tap: $9999."
Maybe you need to actually learn how it works instead of deferring to LLMs that have no understanding of what you are specifically requesting.
Just read the fine documentation.