Perhaps this might seem like a basic question, but why has it taken so long for processors to support AV1, given that it has been out for years?
Compared to other standards in streaming media, I'd say that AOMedia has found adoption a lot quicker. h265 (HEVC) was all but DoA until years after it's introduction Apple finally decided to embrace it. It is still by no means ubiquitous, mostly due to patent licensing, which significantly drives up the price of hardware in single digit dollars price range.
Anecdotally, consider that Apple's HTTP Live Streaming protocol (till version 6) relied on MPEG2TS, even though Apple lay the groundwork for ISO/IEC 14496-12 Base Media File Format aka MP4. The reason was that chips in the initial Iphones had only support for h264 using mpeg2 transport streams, and even mp4 -> mp2 transmuxing was considered too resource intensive.
Really? Muxing generally doesn't require one to decode the actual data - merely shuffle around blocks of data with memcpy() so should be really cheap.
So indeed, repackaging mp4 <-> mp2 containers is pretty trivial. Nevertheless, Apple initially choose mpeg2TS because it conveniently allowed them to shove the reassembled media segments straight into the dedicated AV chip.
This sounds like you might be confusing that MPEG2TS might have something to do with the video encoding instead of it solely being the way the video/audio elementary streams are wrapped together into a single contained format. The transport stream was designed specifically for an unreliable streaming type of delivery vs a steady consistent type of source like reading from a dis[c|k]. There is nothing wrong with using a TS stream for HLS that makes it inferior.
Not wrong, but a bit surprising. As you mention, transport streams are designed to operate over unreliable connections (like satellite or terrestrial transmission). Reliability is not an issue with with HTTP or TCP.
Other than being archaic, some disadvantages are that TS has somewhat more overhead than (f)MP4 and poor ergonomics for random access / on the fly repackaging caused by continuity counter, padding, PAT/PMT resubmission timing, PCR clock.
If it were up to me to design a streaming protocol like DASH, HLS, Flash, or SmoothStreaming I'd instantly choose mp4 (or plain elementary streams). I wouldn't even consider TS or PS unless some spec forced me to.
We seem to be confusing what reliable means here. Yes, HTTP/TCP can reliably transmit data in the fact that if packets are missed they will be resent so that you can be assured that the data will eventually be delivered. However, that doesn't do well for real time streaming of data that is necessary to be received in order. That's why UDP was made.
>If it were up to me to design a streaming protocol like DASH, HLS, Flash, or SmoothStreaming I'd instantly choose mp4 (or plain elementary streams). I wouldn't even consider TS or PS unless some spec forced me to.
Well, it's a good thing we didn't have to wait for you to come around and design a streaming protocol and we've been able to use it for the past ~20 years with the technology that was available at the time. Perfect is the enemy of progress.
Sure, but UDP was not part of the HLS draft https://www.rfc-editor.org/rfc/rfc8216
> Perfect is the enemy of progress.
Touché ;-)
What I meant to point out as odd about Roger Panthos and co.'s decision to build HLS on top of Transport Stream containers is that Apple had already laid the foundation for MP4 with QuickTime.
Since HTTP live streaming was never about anything but HTTP, container capabilities like auto-synchronization offered by mpeg2TS were moot. It would therefore seem logical for Apple to build HLS upon what they already had with QuickTime + iso2 fragments. That was more or less the route Adobe/Macromedia had taken with Flash streaming.
Yet, the choose mpeg2TS (initially only muxing AAC and h264). The reason, historically seems to have been driven primarily by the capabilities of the iphone hardware which supported this out of the box! Separate transport streams for audio and video, WebVTT, elementary stream audio were added much later, and fragmented MP4 was introduced only once HEVC was bolted on.
I'm all for favouring what exists over what's perfect; it's just odd that Apple choose to (initially, at least) regress to 90s technology while rest of world had already adopted superior container.
(Lack of) a compelling use case certainly played a role. Sure, reducing bandwidth is in itself a noble (and potentially profitable) goal but why fix it if it ain't broken? Ie.: the h264 infrastructure for HD was already there, working fine.
Another factor was that HEVC was full of new patents and the patent pool licensing costs hadn't yet settled.
Well, there's now H.266 as well:
https://en.wikipedia.org/wiki/Essential_Video_Coding
https://en.wikipedia.org/wiki/LCEVC
Note EVC and LCEVC are two entirely different approaches to codecs despite the similar names.
No? You're talking in terms of PC / phone hardware support only. HEVC was first released June 7, 2013. The UHD Blu-ray standard was released less than 3 years later on February 14, 2016 - and it was obvious to everyone in the intervening years that UHD Blu-ray would use HEVC because it needed support for 4k and HDR, both of which HEVC was specifically designed to be compatible with. (Wikipedia says licensing for UHD Blu-ray on the basis of a released spec began in mid 2015.)
The licensing shenanigans of H265 was a big motivator for creating AV1, a royalty free codec.
The person who benefits from a more efficient codec tends to be netflix/youtube (lower bandwidth costs), and they are far far removed from the chipmaker - market forces get very weak at that distance.
Netflix doesn't benefit since their catalog is orders of magnitude smaller.
Luckily the effort behind the David software codecs kept up the rollout momentum.
Well the basic answer is that, making an efficient hardware encoder and decoder, within power budget and die space, all while conforming to standard because you wont have much of a chance to correct it, and implementing it into the SoC design cycle which is and always has been at least three years minimum, is a lot harder than most software engineer at Google and AOM would thought.
HEVC was finalized in 2013. AV1 was finalized in 2018, and has just finally started getting a robust ecosystem of software and hardware.
Apple ships probably more devices than anyone, and given that the patent pool is huge as mentioned odds are overwhelmingly that it costs them money to support HEVC / HEIC, not the reverse. That theory also is dubious.
Remember when everyone was yammering for VP8 support? Then it was VP9 support? Now it's AV1. Sometimes it takes a while to shake out. By all appearances AV1 is a winner, hence why it's finally getting support.
This is a nit that doesn't negate your main point: Apple may ship more complete devices than anyone, but Qualcomm makes up significantly more of the SoC manufacturer market share[1] at 29% vs Apple's 19%
[1] https://www.counterpointresearch.com/insights/infographic-q2...
Basically, it was the push to 4K (and especially HDR) that caused HEVC to roll-out. In 2016 4K Blu-rays started coming out and they were all HEVC 10-bit encoded. It took a couple more years before dedicated streaming devices and lower-end smart TVs bothered to start including HEVC support as standard because at first 4K content was uncommon and the hardware came at a premium.
Now that it's mostly the de-facto standard, we see HEVC support in basically all streaming devices and smart TVs.
AV1 didn't have any sort of resolution change or video standard change to help push it out the way HEVC did, so it's basically rolling out as the parts get cheaper due to pressure from streaming giants like Google and Netflix rather than due to a bottom-up market demand for 4K support.
No AV1 videos -> no pressure to add hardware support -> difficult to justify encoding videos in AV1
I don't know how that can be viewed as a good thing.
Roku's latest devices to support AV1, so I guess either the price came down, they struck a deal, or Roku just lost to the market pressure after Netflix pushed for AV1 as well.
Intel, AMD, Nvidia, and other ARM chipmakers for phones, TVs, streaming sticks and such were quicker to pick it up.