Replacing HLS/Dash – Live Mass Fanout with Media over QUIC
quic.video
quic.video
LLHLS is darn good, but it's not as good as WebRTC-powered technologies have shown from a latency perspective. Whether that's required for particular use cases is really pertinent to the case. For non-interactive feeds, it's not really a value add.
That said, the biggest quality improvements MoQ has are how it recovers from a poor connection issue, which happens frequently even on the best of consumer networks and Internet connections, let alone on poorer quality WiFi and cellular networks.
Agree on no segments. WebRTC CDNs are built differently because of this. Your infrastructure looks like a tree, the nodes at the edges are what viewers connect too. One design is [1], but you can have others.
[0] https://blog.livekit.io/an-introduction-to-webrtc-simulcast-...
[1] https://blog.livekit.io/scaling-webrtc-with-distributed-mesh...
But isn't client-side adaptive bitrate switching the point of HLS and DASH? Given that server side software (encoders, live and VOD origins) and clients (shaka, hls.js, dash reference client, set top boxes, smart TVs) exist ... doesn't it make more sense to set up http/3 and have it all in a well established standard?
Apple has committed to adding it, and there's already been some work on it in WebKit, but we don't know when it will make its way into a Safari release yet.
As for maintaining existing standards, it depends on your goals. If you're not trying to push boundaries then absolutely, HLS/DASH is great for quickly getting off the ground. But if you're looking for something in-between HLS and WebRTC then MoQ is compelling.
Re-transmission of old frames is the opposite what you want for live streaming. If you can't re-generate the frame from forward error correction, then you're shit out of luck. if you're hitting network throughput limits, wasting data on transmitting frames you cant display is not a good use of a limited resource.
Wait, what? Is MPEG-DASH really prohibited by Apple for native applications? Or do they just mean "HLS or equivalent; don't do progressive downloads"?
What about applications that can't pick their delivery format, like web cams? Does this only apply to VOD-like use cases?
When/If they notice you're not using HLS instead of some other solution like MPEG-DASH, they will raise a stink and force you to fix it.
I know this because I had the privilege of changing a video playback stack to be 100% HLS over a weekend for a fairly big name app when it became a Pri 0 issue thanks to Apple noticing the app hadn't been using HLS 100% of the time during a minor update.
There's also verbiage requiring a laughably low bitrate stream as a fallback. Our users would file bugs when we fell back to that stream instead of audio only since we had a rich audio only mode they preferred given that the video at whatever bitrate (192kbps?) for our content was nothing but a colorly mess.
This was years ago, the rules are still in the app store, but I have no idea how well they're enforced anymore.
However there are some claims in that article that bothered me:
> if you’re watching 1080p video and your network takes a dump, well you still need to download seconds of unsustainable 1080p video before you can switch down to a reasonable 360p.
A client can theoretically detect a bandwidth fall (or even guess it) while loading a segment, abort its request (which may close the TCP socket, event that then may be processed server-side, or not), and directly switch to a 360p segment instead (or even a lower quality). In any case, you don't "need to" wait for a request to finish before starting another.
> For live media, you want to prioritize new media over old media in order to skip old content
From this, I'm under the impression that this article only represents the point of view of applications where latency is the most important aspect by far, like twitch I suppose, but I found that this is not a generality for companies relying on live media.
Though I guess the tl;dr properly recognizes that, but I still want to make my point as I found that sentence not precize enough.
On my side and the majority of cases I've professionally seen, latency may be an important aspect for some specific contents (mainly sports - just for the "neighbor shouting before you" effect - and some very specific events), but in the great majority of cases there were much more important features for live contents: timeshifting (e.g. being able to seek back to the beginning of the program or the one before it, even if you "zapped" to it after), ad-switching (basically in-stream targeted ads), different encryption keys depending on the quality, type of media AND on the program in question, different tracks, codecs and qualities also depending on the program, and surely many other things I'm forgetting... All of those are in my case much more important aspects of live contents than seeing broadcasted content a few seconds sooner.
Not to say that a new way of broadcasting live contents with much less latency wouldn't be appreciated there, but to me, that part of the article complained about DASH/HLS by just considering the ""simpler"" (I mean in terms of features, not in terms of complexity) live streaming cases where they are used.
> You also want to prioritize audio over video
Likewise, in the case I encountered, we do largely prefer re-buffering over not having video for even less than a second, even for contents where latency is important (e.g. football games), but I understand that twitch may not have the same need and would prefer a more direct interaction (like other more socially-oriented media apps).
> LL-DASH can be configured down to +0ms added latency, delivering frame-by-frame with chunked-transfer. However it absolutely wrecks client-side ABR algorithms.
For live contents where low-latency is important, I do agree that it's the main pain point I've seen.
But perhaps another solution here may be to update DASH/HLS or exploit some of its features in some ways to reduce that issue. As you wrote about giving more control to the server, both standards do not seem totally against making the server-side more in-control in some specific cases, especially lately with features like content-steering.
---
Though this is just me being grumpy over unimportant bits, we're on HN after all! In reality it does seem very interesting and I thank you for sharing, I'll probably dive a little more into it, be humbled, and then be grumpy about something else I think I know :p
> A client can theoretically detect a bandwidth fall (or even guess it) while loading a segment, abort its request (which may close the TCP socket, event that then may be processed server-side, or not), and directly switch to a 360p segment instead (or even a lower quality). In any case, you don't "need to" wait for a request to finish before starting another.
HESP works like that as far as I understand. The problem is that dialing a new TCP/TLS connection is expensive and has an initial congestion control window (slow-start). You would need to have a second connection warmed and ready to go, which is something you can do in the browser as HTTP abstracts away connections.
HTTP/3 gives you the ability to cancel requests without this penalty though, so you could utilize it if you can detect the HTTP version. Canceling HTTP/1 requests especially during congestion will never work through.
Oh and predicting congestion is virtually impossible, ESPECIALLY on the receiver and in application space. The server also has incentive to keep the TCP socket full to maximize throughput and minimize context switching.
> From this, I'm under the impression that this article only represents the point of view of applications where latency is the most important aspect by far, like twitch I suppose, but I found that this is not a generality for companies relying on live media.
Yeah, I probably should have went into more detail but MoQ also uses a configurable buffer size. Basically media is delivered based on importance, and if a frame is not delivered in X seconds then the player skips over it. You can make X quite large or quite small depending on your preferences, without altering the server behavior.
> But perhaps another solution here may be to update DASH/HLS or exploit some of its features in some ways to reduce that issue. As you wrote about giving more control to the server, both standards do not seem totally against making the server-side more in-control in some specific cases, especially lately with features like content-steering.
A server side bandwidth estimate absolutely helps. My implementation at Twitch went a step further and used server-side ABR to great effect.
Ultimately, the sender sets the maximum number of bytes allowed in flight (ex. BBR). By also making the receiver independently determine that limit, you can only end up with a sub-optimal split brain decision. The tricky part is finding the right balance between smart client and smart server.
My point was more about the fact that I found the article unfairly critical towards live streaming through DASH/HLS by mainly focusing on latency - which I understand may be one of the (or even the) most important points for some use cases, but weren't much on the cases I've worked on, where replacing DASH would be very complex.
This was kind of acknowledged in the tl;dr, but I still found the article unfair at this level.
Article mentions droppping from 1080p to 360p like it's a feature, but I would say its horrible to do that...
If I'm watching a movie or listening to music I don't want random switches to poor quality or lower resolution and associated glitches.
I'd much rather see the spinning wheel and complain to my provider that I'm not getting what I paid for or check what's going wrong.
I can understand various resolutions and codecs based on user preferences and devices but that's a fixed choice.
Am I in the minority here?
As someone who's worked in video delivery, I can't overstate how much the majority prefers not to have stutters in their video while sitting on a subway/bus/train/massive SUV.
Personally I use yt-dlp with mpv and have the buffer set very large. This allows me to pause the stream when I take a toilet break or similar. When I come back I can also skip ahead through ads or idle times on the stream, or watch at increased speed to catch up if needed.
I was thinking of experiences on YouTube where it refuses to buffer enough to allow playing the whole video, leading to inevitable stutters.
A full minute of buffering would only give about 5% more bandwidth for a typical 22 minute TV show. That’s not going to transform it from “something unwatchable” to the higher bitrate video you want.
It will, however, cause a large majority of people to give up on the stream. The number of people who would actually prefer 1 minute of buffering for negligibly higher quality is almost nonexistent.
This is why buffering is futile: Once a stream gets to the point of stopping to buffer, the mismatch between available bandwidth and the encoded bitrate is just too great to reasonably overcome.
For your point, an example is viewing 1080p on 64kbit/s is unreasonable and nothing will fix it.
For OP's point, I can imagine viewing 1080p (10Mbit/s) on a 1Gbit/s connection means a little buffer will solve the need for ABR as random packet loss or minor drops can be recovered with retries seamlessly as opposed to dropping quality to ABR.
So buffering is futile only when bandwidth is not sufficient, and even there its questionable, perhaps user might just want to download and view after that's complete rather then have poor quality.
YouTube is very aggressive about switching to bad quality if it thinks the connection is bad, which happens often due to transient issues on cell networks or occasionally wifi. All of a sudden a video will drop to 240p, for instance, reminding me "ah, it reset itself to 'auto', I never ever want 'auto'".
A minute of buffering isn't enough to make an incredibly slow connection fast. But a minute (or potentially less) of buffering is enough to paper over transient connection issues.
I'm not surprised it's a similar tale with video delivery.
In my experience, what frustrated me most was predictable stalls every 10s or so. I don't see that much any more.
More annoying even now are the constant "Hey moron, is your video stuttering? Try lowering the quality!" 4head tapping messages I see on Twitch or YouTube when their CDN is taking a shit. I pay good money for 1G/1G symmetric, my PC is 6 feet from my wireless AP. If you can't feed me 6Mbps of live stream at me without stuttering I don't think it's a me problem.
If MoQ means an end to or improvement that terrible ABR experience, it can't come quic enough.
[0] I say opt-out not opt-in because the audience for ABR is people who wouldn’t know where to find an opt-in button or wouldn’t know that it exists.
I was having issues that even if my player indicated it was playing 1440p, it clearly was at most 640p or something like that, complete potatovision. Re-clicking the 1440p option would fix it, and this extension basically does that for you.
Good luck on mobile, though lol
Probably. I used to work in broadcast, so I have been conditioned to seek out the highest quality video I can find. So like you, I will accept buffering for higher quality playback.
However I still see people who force 4:3 into 16:9 because it fills the screen, and people who still have motion smoothing on their TV so everything looks like 60fps until the stream breaks.
however on mobile, and I just want to see the shitty politician say the shitty thing.
I think most people want decent audio with mostly moving pictures. Most people can't see HD at TV distances anyway, so I suspect so long as it sounds good, most people don't notice too much.
It’s also worth mentioning that Peter Jackson’s The Hobbit was shot at double FPS (48 if I recall correctly), and was regarded by many as looking “off”. Based on the fact Hollywood has not made any high FPS movies since, it seems that it was generally regarded as a failed experiment.
I wonder if new display tech is contributing to this becoming ever worse of an experience? With the tv snapping to the new frames practically instantly and then waiting for 50ms with no visible changes, motion blur baked into the movie can no longer hide the judder effectively.
I haven't seen The Hobbit in cinema, so I can't comment on that experience. Maybe they just did it wrong somehow, which would be a shame.
LCD/OLED follows a "Sample and Hold" paradigm, whereas CRT/Plasma don't have the same limitations.
Film projectors would close the shutter while the film moved, and might open the shutter two or three times per frame; I wonder if you'd do well with black frame insertion than some of the modern displays do? Personally, it seemed pretty awful for me on a samsung qled, maybe it's more useful on OLED; but I also dislike motion smoothing, so clearly we have different taste.
I will agree with you that panning shots in 24 fps aren't great. I think there's some rules of thumb for cinematography to reduce audience discomfort, but not everyone follows those.
And while 48 fps is a major improvement from the prehistoric 24 (apparently it was chosen because of the power grid a hundred years ago??), it's still a far cry from actually smooth motion at 144 and above fps.
If rendering the whole movie at this frame rate is too difficult, it would be quite interesting to explore exporting the additional buffers (velocity, depth, etc) that allow DLSS 3 frame generation to work near perfectly in games. Can cameras even capture these?
Creating depth maps of existing images/videos is one of the interests of the field of Computer Vision, it is not a solved problem.
When I watched Avatar 2 in IMAX 3D I didn't notice the drop to 24, was that during talking/low motion scenes only? I'm sure if I looked for it I could see it, but I went into the movie not knowing that refresh rate switch occured.
We have 8k/120Hz cameras now, although I'm not too well versed in how increasing the refresh rate affects how the camera captures light, and what effects that has artistically.
Lidar is noisy, and doesn't have a 1:1 pixel mapping between image pixel and depth pixel.
Even when we had stereosopic recordings for 3D the depth map we could generate was always a bit shit. Granted things have moved on in the last few years.
> We have 8k/120Hz cameras now, although I'm not too well versed in how increasing the refresh rate affects how the camera captures light, and what effects that has artistically.
If its a RED it has shit colour sensitivity. however, generally running cameras at higher framerates require more light. but the shutter angle is generally faster than the frame rate (when you record video on your phone outside in the sun, the shutter speed is 1/1000th of a second, even though its doing 60fps)
The same way I disliked the remakes of Dune and Bladerunner. I don't know exactly how to put it, but despite the gorgeous graphics they felt rather empty to me, like a facade in theme park, with not much behind it.
OR like Nina Hagen once screamed: 'TV, Teeh-Vau, Ich glotz Teeeh-Vaauuuuu... (<-imagine german pronounciation there) ...Alles so schön bunt hier...'
You would think that a device whose sole purpose is to divine a picture from wagging pixies in the air and make the little lamps twinkle in a convincing way would be able to do it by now.
But I have yet to find a "smart" TV that can handle it decently, even well into the mid range it's still a bit hit and miss. Get a cheap decoder and plug it into any mid range laptop and the pacing is leagues better.
Do they just use the absolutely worst SoC that still turns enough product to pay the bills, or what?
SuperHighDef (https://en.wikipedia.org/wiki/Ultra-high-definition_televisi...) is absolutely brilliant. I saw it live at a broadcast show in 2012.
However motion smoothing is an aberration, as is the stupid saturation and "artifact removal AI" that basically applies a box blur to everything, then applies sharpening to make up for the slow LCD response.
The thing that fucks me off about motion smoothing is that its so unreliable, its smooth for 8 frames, then breaks and comes back.
However, I think the reason why most people hate high framerate video is that is associated with cheap TV. If you're young and have never seen crappy low budget TV, I could see that you don't have that association.
The reason why 24FPS feel "expensive" is because its associated with high budget movies(so does high colour kodak feel).
That's unfortunately true, even on the best TVs. But for me that is merely slightly jarring, while panning cameras at 24 fps are completely unwatchable. So I begrudgingly use it and hope that some day they will realize movies in 2023 should not be using the same frame rate they had due to technical limitations 100 years ago.
A true 120 fps movie would be so, so much better. Just like how I am playing all games at 4k / 120 fps on the TV.
Something I've been meaning to explore is pre-processing a movie with RIFE, which produces a much better result than the builtin motion smoothing since it doesn't have to run in real time on a subpar processor. Though it takes quite a while even with an RTX 4090 and you need the actual frames so can only do it with pirated movies from blu-ray rips, which is annoying.
I'd be fairly sure 360p is well below it today.
It would be interesting to "see" the packets on screen arriving and decoded with no error correction or buffering (maybe with reconstruction of missing squares...) , like an analog kind of signal : no latency, full available bandwidth, no waiting (but sometimes... less than ideal quality).
If I'm watching a live stream, I'll take whatever quality currently works. Waiting to buffer it would mean I'm behind everyone else...
Yes, and it's not even close. Metrics are clear - stutter/loading will cause many more people to close the window vs lower quality.
> I can understand various resolutions and codecs based on user preferences and devices but that's a fixed choice.
What are the odds that many of those users who set a preference and forget about it, now are wondering why youtube etc. never works at their coffee shop or when in the back yard where the WiFi is weak? The majority of video on the internet is viewed from wireless networks which always have a chance to vary in quality
You are deeply in the minority.
People hate buffering because it’s rarely a 1-time event. Once a video starts buffering, it’s usually a sign that something is wrong and it’s not going to continue happening. People have internalized this and will give up quickly once buffering happens.
You’re also assuming that the issue would be with your internet provider and that complaining to them will resolve it. Neither of those assumptions are true most of the time. While there are cases of degraded internet downlink performance, the more common failure modes involve people’s WiFi. Borderline connection quality can turn into interruptions when their nearby neighbor starts a download on the same WiFi channel, or if someone in the same house starts using the WiFi at the same time.
I like getting feedback. I don't like when platforms try to smooth out the rough edges invisibly... they often end up in me experiencing breaking problems without my realizing I am experiencing breaking problems. It extends so much farther than having the choice to force a video feed resolution.
I don't like when voice and video channels "seamlessly" pitchshift and speed-up/slow-down (at least, without indication). I already find it unacceptable how aggressively most platforms like to compress sound in their audio codecs, presumably in the name of saving bandwidth.
Slack application going 1000% out of their way to never let you know that it has a connection drop, so you never know if it re-established comms when you know your net dropped out for a moment, or if you need to hard refresh with Ctrl+R, because you heard the Slack notification on the phone right next to you, but you can't tell whether the desktop app has received the latest messages (often, it hasn't).
In all seriousness, I would love something where my computer lcd subtly shades red or blue along the edges of the display depending on when I'm getting network jitter/latency and when it's catching up, and things like that. How cool would that be?
Yes, actually.
I know that I'm talking about apples vs oranges, but how long have we been waiting for IPv6 to take over from IPv4? I don't see QUIC taking over from TCP any time soon.
HTTP/3 uses QUIC, so there’s already a substantial amount of production traffic going over it.
But unlike IPv6 there's few barriers to adoption of QUIC where it really pays, so I expect that QUIC will be adopted by a lot of applications pretty quickly.
lol the telcos will just throttle you, they don't care that your twitch stream constantly pauses to buffer. lower bitrate streams make the content playable, period.
AT&T customers are less likely to complain if AT&T throttles a HLS broadcast, as the quality will just be lower.
What do you mean? Are you saying you can't serve content via QUIC with a certificate issued by lets say letsencrypt certbot?