How the BBC and ITV are fixing delays on World Cup live streams
wired.co.uk
wired.co.uk
Yup, this sums it up. Now that we're in the second week of the tournament there are less matches to cover and the radio becomes a good option. Before that, there are too many good matches on outside courts to stick to the main action.
Sports are a good test case, not just for latency but sheer demand. I worry that we'll face some kind of natural disaster in the not too distant future that'll make us wonder why we abandoned the resilience of OTA broadcast.
There have been several attempts at doing the same thing in the U.S., but none of them ever worked out for various reasons. Often it was because of the expense of upgrading the tens of thousands of TV transmitters across the country.
This one died in 2011: https://en.wikipedia.org/wiki/MediaFLO
In a small country with more centralized control of broadcasting and less territory to cover like Japan and Korea, it's much easier.
https://en.wikipedia.org/wiki/Digital_multimedia_broadcastin...
https://www.bbc.co.uk/news/entertainment-arts-44722812
Actors have vented their fury at two audience members who watched England's World Cup penalty shoot-out on their phones in the front row of a musical.
Titanic The Musical actor Niall Sheehy said the women "not only followed the penalty shootout on their phone, but also said 'yesss' on each goal scored".
He declared: "You are the most ignorant audience members I have ever had the misfortune to perform in front of."
Rude people are always going to be rude, and we shouldn't worry about the technology that enables them to do so. If they couldn't watch soccer during a musical, they'd just be coughing loudly or rummaging around in their pockets looking for a mint or having a too-loud "private" conversation with the person next to them.
Phones have changed nothing.
Seems to me that theatres (and cinemas) should be putting passive farady cages to block signals coming in, I'd pay extra for that.
They got paid to perform a play. There is no more obligation for the audience members to pay attention than there is for someone to eat all the food on their plate at a restaurant, or use a device they bought every day. If the women weren't disrupting the other audience members then the actors shouldn't have cared (and if they were, that's a problem for the house staff to sort, not them).
If actors understood that it's not the audience who is privileged to watch them act, it's them who are privileged to be paid to play make-believe on stage, the world would be a much better place.
Everything else makes perfect sense to broadcast, but live -- especially zero-latency live -- over unicast http calls? Ridiculous. A shame the BBC's multicast trials never took off.
3.8 million according to https://www.theguardian.com/media/2018/jul/08/england-world-...
I.e. people tried iplayer, saw how crap the experience was (in this niche case), then found a TV or pub with TV to watch it on.
In this case the stream went down in the last 5 minutes of the match, which is still bad, but not as bad as YouTube TV earlier this week!
And its a matter of complexity - its obvious why OTA would have less problems to solve in order to get the same performance and behavior of something more complex.
IP streaming does actually solve a lot of problems. Plus these days OTA transmissions are all digital anyway. So serving that digital content via the internet is a natural progression. Sure things are superficially more complex at the moment but the technology hasn't matured yet.
21 years ago, we learned to hate the word "Buffering..."
I'm not convinced how muliticast works to mobile devices, but it's a painfully edge case. This type of event occurs in the UK once every 4 years (the olympics doesn't pull in these numbers -- except for 2012 when it was hosted in London), and even then only when England do well (so once every 20 years)
That said, ITV latency seemed far shorter than BBC latency, watching ITV player seemed acceptable. Apart from the lackluster commentary and the adverts.
* when the broadcaster screws up their RDS signal after a traffic report leaving you stranded on the wrong station.
* Or FM stations that run at different frequencies from different transmitters (this used to be a massive problem when driving long distances even just 20 years ago but now even the analogue car radios are digital devices that scan multiple frequencies to hop to the next frequency aired by a given station so you don't experience much cutout).
* Then there is the obvious issue of "dead zones" where the signals cannot reach (tunnels, some industrial locations, etc).
* Digital radio also suffers from frequent, albeit brief, cutouts while driving too but I've found that to have fewer dead zones than analogue.
* Bleeding used to be massive problem with analogue radio as well - this is another problem that was solved when radio started going digital.
So this is with 100 years of research and development into a protocol that requires an order of magnitude less bandwidth and yet most of the improvements have only been in the last 20 years when things like RDS (digital packets in analogue signals) and DAB have been introduced. Given that IP streaming video content isn't that far behind, just imagine how good it is going to be when that technology finally matures.
Wait, is this really a common thing? How would I know if a car radio does this, because I'm 99% sure my car doesn't.
Just like the phones in the UK will, in the not too distant future, move from POTS to a fully packet switched network, so too will television, especially once at least superfast (24Mbps and up) broadband is available to all - there are already rumblings that Sky would like to move to IPTV instead of having to bounce their signals off a satellite.
In this case, to have more reliability in transmission, you have to have fewer choices.
I can't speak to the UK's digital situation, but in the U.S. there are 50 channels available in a theoretical city. Assuming that digital has fewer co-channel issues than analog (most of my TV experience is analog), that means 25 signals. On those 25 signals, some broadcasters squeeze up to 10 channels, which all look like garbage. For a decent picture, it's closer to four — maybe five.
So that means 100 to 125 theoretical OTA channels.
If people could actually receive 125 OTA channels with good programming on them, or even the same content as is offered on cable/satellite, I believe many would drop pay TV in a heartbeat, and wouldn't be forced to also scrounge for content on Netflix and YouTube and Prime Video.
If anyone has information on the theoretical maximum number of TV channels possible in a single location in the UK, I'd like to hear about it.
But in this case we really want actual broadcasts. Talk about square pegs and round holes...
Good read around the steps and impact around buffering introducing delay. My programmer brain said “sounds like the receiver should control flow into its encoder”... so is it that it’s too conservative a buffer, or the encoding algorighms actually need that much buffer to get economy?
Even now it's digital here in the UK I believe each of the 5(?) muxes can only support tens of channel each.
Airwaves TV might scale easily but can only send a proportionally small amount of information, but the internet send a vastly larger amount of data per second, but struggles when everyone wants the same data at once.
> It's instant
It isn't. There are still latency delays with digital encoders, transmitter hops, etc even with OTA signals. It's just the equipment has gotten so good over the years that the latency has been pushed right down but it did used to be a lot worse. The same will happen with IP streaming given time.
> scales infinitely
It doesn't. You need geostationary satellites / multiple transmission towers, and all that only provides you a finite number of channels. In fact digital streams are actually much cheaper to scale up, which is why so many broadcasters are moving into this territory (and why I currently have a job)
> kind of beautiful simplicity, technically speaking
Superficially yes. But if you spend any amount of time around OTA broadcasters (as I have) you'll see there is a hell of a lot of complexity hidden from sight. In fact since OTA broadcasts are still digital streams served from content stored in digital formats, a lot of the technology used for internet streaming is the same for OTA broadcasts. So there is an argument for the simplicity of just serving that digital stream via IP rather than radio waves (or are you suggesting we go back to analogue as well? Because that also introduces a lot of complexity but of a different nature, like having to giant robotic cabs filled with hundreds of magnetic taps that all have to be wound to the right point; and a dependency on the postal service delivering those tapes since you now cannot just download assets from the content owners).
I'm not saying there isn't a lot of new complexity with IP streaming but I do think it's more pronounced because:
1/ we work in IT so follow the technology more closely than we do for OTA broadcasting
2/ it's still a relatively young industry so there are a lot warts that haven't been cleaned up (much like how OTA broadcasting used to be a lot less polished when the technology there was still evolving).
Broadcast television isn't particularly unusual, surely. Millions of people watch live, primetime TV every day.
However the use case of live sport - especially football in the uk - is massively different. This means the platforms have to scale to 20 million plus live streams (that's say 100tbit and 5 second latency) for a once in 4 year event.
I couldn't say how close they are to getting that working - nor if it even is possible - because I don't work on that specific side of things. But I do know there used to be multicast networks over the internet back in the 90s (IIRC David Bowie even streamed one of his concerts over it) so it strikes me as a solvable problem.
However multicast struggles over wifi and 4g.
A CDN edge box in each exchange would probably solve the problem.
I'm not so sure. When I had analog over-the-air TV, in a non-capital city, I remember quite a few times when most of the channels would go down during a storm.
"Scales infinitely" is underspecified. Over-the-air broadcast scales well with the number of people who want to receive a particular broadcast. It scales very poorly on some other dimensions:
- Over-the-air suffers from congestion when many parties want to send a broadcast simultaneously. There's only so much spectrum. In a wired network, you can just add more wires.
- Over-the-air doesn't scale well at all when people want to receive the same broadcast from farther away. In a wired network, you can use longer wires, or relay data from one wire to another wire. (Relays will work for wireless too, but they make congestion that much worse.)
Unfortunately multicast doesn’t work across the internet. It’s typically filtered or ignored I think.
Not sure why multicast wouldn't work from behind nat, assuming the router supported it.
I had an old blog page about how it was designed but I took it down. May be worth revisiting it. That said the general public prefer unicast so they can watch it when they want, which renders multicast a moot point.
Nagra sell a system that does it to ISPs around the world and have done for at least 10 years. I'd been hoping we'd see more of that in the UK, but there has been virtually zero movement probably largely because of our patchy broadband service. I'm assuming the Sky Q boxes are heading in that direction eventually.
We know there is a roughly 5 to 6 seconds delay from Real life to TV Broadcast. ( Or may be from Betting company to TV Broadcast ) [1]. Which should have been enough time for OTT to catch up to OTA. Assuming if they could allow OTT to have 1.5 second head start of OTA.
And to be honest I am very much surprised how unprepared all these OTT players were ( If you live in a country that was well prepared and offer low latency streaming, please forgive me ). As if they just discover 30+ sec delay in OTT delivery is going to be a big problem in World Cup. And this isn't as complicated as a Low Latency Live Streaming to Worldwide audience which requires much more complicated Edge and POP measurement and design, they are basically streaming to audience within their broadcasting right which is all regional.
I have a hard time understanding why we cant do sub 8 seconds delay delivery today ( Compared to TV, not real life ). I think this is mostly due to manager and marketing department decide to throw in OTT as an added features for their customers to enjoy / upsell their package in the last minutes without fully considering, investing and testing their tech. We should have been aiming for 1.5 sec this World Cup. Not Next.
P.S - I do agree with another comment about OTA broadcast. I just wish our phone could receive those signal without much compromise. Instead it seems the world is moving towards IP based everything.
[1] https://blog.benjojo.co.uk/post/beating-the-broadcast-delay-...
https://www.bloomberg.com/news/articles/2018-07-11/youtube-t...
https://news.sky.com/story/world-cup-fans-fuming-as-bbc-ipla...
1 minute 40 -- that's the time from the neighbours shouting and the goal going in -- on iphone, on mac/chrome, and on LG TV
It's hilarious.
I was really looking forward to the final, especially if it went to penalities :(
This article talks about the 'broadcast' over internet, which is even worse.
now one thing i'm impressed is that the internet transmission from SporTV had basically zero lag when compared to the TV one.
Don't get me on satellite latency 20 years ago where people used to make money on betting on horse races where the data feeds in betting shops were about 40 seconds delayed...
Satellite adds far than a second.
Anecdotally, YouTube TV had ~15 second delay and Fox Sports Go had ~50 second delay compared to what I see on the cable TV.
https://www.fomopop.com/guides/live-tv-streaming/specs#live-...
Most of the services were around 30 second delays, with the best being around 15 seconds and the worst around 60 seconds.
At this point there needs to be a technology shift if its going to get any better than about 10 seconds.
It even varies by device. iOS defaults to 10 second chunks, but you can reliably get it down to 6 seconds. You generally need to store at least 2 chunks or you'll have bad buffering.
To be fair, apart from the lag and that crash at the end of one game, the BBC stream was a rock-solid HD, and it almost never buffered. ITV instead was low-res crap, stuttered all the time, and even had compulsory ads. Both streams got worse as the tournament neared the end, of course; in the first rounds the latency was only a few dozen seconds.
An idea which makes me incredibly curious as to how well Twitch would handle that type of traffic!
[PS Before anyone mentions RTP, this is a known problem with RTP - see: https://www.cs.columbia.edu/~hgs/rtp/faq.html
"RTP does not ensure real-time delivery. So how come it is called a real-time protocol? No end-to-end protocol, including RTP, can ensure in-time delivery. This always requires the support of lower layers that actually have control over resources in switches and routers. RTP provides functionality suited for carrying real-time content, e.g., a timestamp and control mechanisms for synchronizing different streams with timing properties." ]
Good luck getting the cable companies to agree to a delayed signal.