455 Tb of live streaming were transferred during the Olympic Games Rio 2016 | Hacker News Reader

455 Tb of live streaming were transferred during the Olympic Games Rio 2016

leandromoreira.com.br

147 pointsdreampeppers9994 comments

> Globo.com only had rights for streaming the content to Brazil... with peak of 433K simultaneous users

This is a neat writeup but the bigger point here is highlighted by the opening statements. While i get the massive revenue channel the olympics provides, i can't help but feel that the media stranglehold over the events has gone too far.

The olympics represent one of the only instances of global unity through competition left in the world. Much the same way the world cup brings small and proud nations into the global stage, the olympics has the same power. Unfortunately, it is easier for me to find 20 year old simpsons clips on youtube than it is for me to find a video of Bolt's 100m heat from last week. Who benefits from this apart from NBC? I can list approx. 7.4B people who don't.

As a lover of capitalism this all leaves me conflicted. But i would point to the olympic coverage as the instance of corporations going too far with their constricting of media sharing.

Meanwhile, CBC of Canada broadcasted all olympic games for free. They had peak of 27 simultaneous channels transmitting competitions with excellent quality.

Anyone with a Canadian IP could watch, I personally used a VPN because I don't have TV cable here in Brazil.

So did the BBC. They broadcasted for free multiple channels of Olympic events to any and all British IP addresses. I also used a VPN because I don't have cable TV.

I have no idea why Olympics events are not freely broadcasted in the US.

Because the US does not have well funded public television as the UK (the BBC) or Canada (the CBC) do. In the US, sadly, it is to the highest bidder that the broadcast & streaming rights go to. NBC paid $1.23 Billion for rights to the 2016 Summer Olympics, an astronomical amount.

They only provided streaming to those with cable packages above what is considered 'basic'.

hotstar.com broadcasted Olympics free of cost in India. They aren't public funded.

Channel 7 here in Australia broadcast on 3 channels for free (with ads). They also offered free streaming on mobile devices for a range of sports and had a $20 deal for complete access to all content.

That said, they really annoyed me with the way they handled the time difference. They would open their prime time coverage with spoilers of all the previous day's events and then tell you which replays you could watch on the other channels later....

And the whole time the ABC news (Australian one) live stream was constantly being cutoff with "due to licensing restrictions blah blah blah". Sometimes for several minutes, I don't know if that was because of particular segments or just someone forgetting to switch it back on.

I pretty quickly went from casual interest to outright hostility to the olympics.

My friend once tried to watch a cycle race on BT sport's on-demand web player, to start the stream he literally had to click through a spoiler as to who had won. He was apoplectic, and has not used it since.

Because the free market has no interest in that.

Same in France, probably most of Europe actually

In German it was broadcasted for free too.

>Unfortunately, it is easier for me to find 20 year old simpsons clips on youtube than it is for me to find a video of Bolt's 100m heat from last week.

So glad I wasn't the only one. For whatever reason the nbc link to the video was never on the front of google.

The ads streamed just fine. None of the actual events streamed after the ads finished.

Perverse that ads always work. I feel like I'm owed some kind of monetary compensation every time I watch an ad and then don't get to see the video I wanted to watch.

This was maddening as well. I had to go through nbcsports app and dig deep to find "Michael Phelps wins gold after thrilling final!" Instead of "200m X stroke final"

Such apps are menace. It seems that these apps cannot even do anything better than simple string search. They should learn a thing: Let Google do the search for them not their app.

A joke for the people who know little bit of programming: Hope they are not using strcmp to find the video file with title "Michael Phelps wins gold after thrilling final!" :(

edit: a typo

Probably untuned MySQL full text search, possibly Solr with all the default settings looking only at titles.

> As a lover of capitalism this all leaves me conflicted.

I don't want to go on a tangent but this is the exact contrary of Capitalism, it's a monopoly: as a consumer I can't make a choice, there is no market.

I feel the word monopoly gets way overused. "There is only one company with a train from Town A to Town B! It's a monopoly!" Probably not, as if there are other viable ways to travel (car, bus, plane, boat), then there are substitutes.

The olympics are no different. As a form of entertainment, there are countless substitutes. If the Olympics are boring, people won't watch. Even as a form of international cooperation, there are many substitutes that compete with the Olympics for the time, money, and attention of nations.

There is a market.

Just like public utilities don't have a monopoly on the provision of power, because, hey, every home-owner can buy a diesel generator. With the level of substitution you are describing, you've made the word monopoly meaningless.

The regional broadcasters for the Olympics have a monopoly because of their economic power in setting the price for their good. Other than circumventing copyright or using a VPN, there is no competitor that will provide you with the Olympics for less money. Random youtube videos are not a substitute for watching an Olympics broadcast live.

For many routes, planes/trains/coaches are substitutes, so it's meaningful to say that trains are not monopolies. How do we know? Because when train tickets go up in price, there's a measurable effect on the number of coach tickets bought.

Your example (household electricity vs. diesel generators) doesn't work in the same way. If domestic electricity prices in London were to go up by 25℅, the number of diesel generators and amount of diesel fuel sold would not budge. Because diesel isn't a reasonable substitute.

If anyone wants to know more, search the internet for 'cross elasticity of demand' and 'definition of substitute'.

That's like saying HBO has a monopoly because only they can broadcast game of thrones.

If a product must have a perfectly identical substitute to not be a monopoly, then more or less all companies have monopolies. I feel that position is untenable.

Public utilities are regulated, which require them to make their case in front of a judge for price changes.

Much of the Olympics wasn't live for the west coast even if you watched on TV.

So you're saying that the alternative for the Olympics, a international gathering of most countries on the planet, is comparable to let's say... watching a basketball game? I think your analogy is very inaccurate.

Would you go on a tangent with me? Monopoly is completely in-line with capitalism, consumer choice is only guaranteed in a happy capitalist state.

A lot of the things people like to attribute to capitalism are really aspects of a perfect market. More off a convenient fiction for framing Econ 101 discussions on than something that actually actually exists. In some cases useful as a first order approximation.

It's the economics equivalent of frictionless spherical cows in a vacuum.

Except that while said cow is absurd at the face of it, economics 101 is passed off as gospel.

I feel the same. I've been travelling during the past month. Accessing video content from the Olympics has been extremely hard.

I think the IOC might be shooting themselves in both of their feet here. If they don't open open up, they'll some day find that they're suddenly irrelevant.

By the way, during the 2012 London Olympics the IOC and Google struck a deal to stream the like 20+ HD channels live to 64 territories:

"The live-streaming on the IOC’s YouTube channel will provide exclusive digital access to the London 2012 Olympic Games in territories where digital broadcast rights have not already been acquired by the IOC’s broadcast partners"

https://www.olympic.org/news/ioc-to-live-stream-london-2012-...

https://youtube.googleblog.com/2012/06/ioc-to-show-live-cove...

I was in one of those regions at the time. It was awesome.

I think the IOC might be shooting themselves in both of their feet here

They might get a piece of the vig kicked back to them, but by the time the games started the IOC already had their payday.

I dunno, I found video of Usain Bolt's 2016 100m in less than a minute of looking around on nbcolympics.com. Though I'd have to imagine that that content is geofiltered, since AFAIK NBC only has streaming rights within the US.

NBC paid the IOC something absurd like $12B for exclusive US media rights for the games through 2032. Territorial broadcast rights agreements are a huge source of funding for the IOC. It's hard to imagine how they'd maintain these revenue streams if they abandoned the exclusivity aspect of their media agreements, which likely implies that the games would suffer in some way.

I completely skipped watching the Olympics in large part because of NBC which is terrible. So, I have some doubt that 12B is optimal for the IOC and it clearly sucks for the rest of US.

Watching via the BBC is a completely different experience: complete on-demand archives almost every event, time bookmarks for groups/bouts/matches, no annoying commentators unless the event was also being broadcast on a channel, no missing most of the event because we need to watch a manufactured interest piece on the US athlete, and you can go 30 seconds without hearing the name Phelps.

Watching NBC's broadcast, in contrast, made it seem like NBC had no belief that any of the events were interesting in and of themselves.

I hate those sappy interest pieces! Just show me the event.

You are right. They seem to think that the event is not interesting.

I have historically avoided watching the Olympics because I like to watch sports, not drama ;-). Having said that, NHK in Japan did a pretty good job, I have to say. I watched the entirety of the women's rugby 7's semis and finals. Great sports that I wouldn't ordinarily get to watch, and no Japanese participation at all.

I did the same. I tried to find a live stream online, and failed. Everything I tried wanted me to login with cable credentials. Ended up watching it on my broadcast TV for about 10 minutes before giving up, and didn't watch anymore.

Too late now, but you could have purchased a Playstation Vue subscription for one month ($30, one week free trial) which would have given you full access to the NBC Sports app and nbcolympics.com. You can sign up on the web and don't need a Playstation. That's what I did since I don't have a TV subscription and it worked well. The NBC iOS and tvOS app was pretty awful in forcing ads, but the Roku app let you skip mostly skip through them, which is how I did most of my watching.

I wish I knew that before, tried a Sling Tv sub, and that didn't give one access to nbcolympics streaming content. I kept it for the (poor) live NBC streaming of the Olympics, but cancelled it after. Having to watch shows at a fixed time is crazy in the modern age. I watched much fewer sports and the advertisers probably lost out on quite a bit of exposure too.

I wish I knew that before

Which is directly due to NBC being the exclusive licensee for Olympic IP in the US, so the Vue could not advertise itself in terms of providing Olympic coverage. You could have come across one of the many articles and howtos for watching the games without cable, but it never would have been advertised to you.

I watched more this year than ever because of the NBC app on Roku. I could watch any sport live or streamed later. No need to watch the Costas garbage.

"I found video of Usain Bolt's 2016 100m in less than a minute of looking around on nbcolympics.com"

The GP wanted to find "Bolt's 100m heat" which is a different event from the final (athletic heats take place before semifinals). His point was that it is very hard (in the US) to find footage of Olympic events of secondary importance.

If NBC were hosting a fireworks show they'd make the people that didn't pay stay indoors.

Edit: Perhaps the strategy isn't effective, that's all.

And for the Olympics, the ancient games meant to bring people together, I for one couldn't watch 5 minutes of any of it as I am a chord cutter and couldn't stream jack, legally that is.

No one should have a monopoly on the Olympics. I find that rather sad and telling.

London 2012:

He said: "You probably wouldn't be walking in with a Pepsi T-shirt because Coca Cola are our sponsors."

Interviewer Evan Davis asked Lord Coe if he would be able to enter the Olympic Park wearing Nike trainers, to which Lord Coe replied: "Let's put some reality in this. You probably would be able to walk through with Nike trainers."

Lord Coe defended the ban on 'objects or clothing bearing political statements or overt commercial identification' outlined in official guidelines for spectators.

http://www.telegraph.co.uk/sport/olympics/news/9414178/Londo...

>'The olympics represent one of the only instances of global unity through competition left in the world.'

What were the other ones that have presumably ceased to exist?

The World Cup is also probably pretty close too. Maybe Eurovision also (for Europeans under about 35, perhaps).

On the other hand, it did motivate me to find a way to stream NBC Sports. Something I wouldn't have done otherwise.

I cannot stand NBC and their selfish Olympic coverage.

I'm interested in learning what the average "delay" for live-streaming would be? At a high-level RTMP > (Segmenter: EvoStream) HLS > Cassandra < NginX-LUA < HTTP Request (seems to flow model outlined here (From the 2014 World Cup): https://www.nginx.com/blog/globo-coms-live-video-platform-fi...)

I'd assume it's minimum > 20 seconds depending on setup and teardown time for first "chunk" in a sequence to reach cache and be transferred to a user?

TL;DR: The delay is normally a segment length. Somewhere between 2-16s depending on there configs.

Because the system is realtime (live) it _must_ not fall behind live. These systems therefore are always only a single segment behind live. (A segment is an independently decodable chunk of video, usually between 2 and 16s).

Source: I designed the backends, including segmenting, for NBC.

@zbobet2012 makes sense, certainly, you can shrink the chunk-size down to 2 seconds, I wonder if it's not practical to do though wouldn't the practicality of setting up a setup > teardown of a new HTTP connection, fetching the next chunk, etc. cause inconsistency in completing this procedure by the time a 2 second chunk has played out.

I believe people have used WebSockets to push these segments? Since once you have established one TCP socket connection you don't have to setup a new "session" for each discrete segment?

We (Comcast/NBC) run 2s segments for our video streaming service which is... substantially larger than this. High latency mobile networks on clients which don't do keep alive can be _somewhat_ problematic. The best fix for that is just to have a very fast/close CDN edge node.

HTTP keepalives prevent the excessive connection requests. The main issue with HLS distribution is TCP sucks over high latency links, so you have to have edges near your users.

Gotcha! I've also heard of people using WebRTC to reduce latency by setting up a direct-to-user link between the ingest server and the end-user? Any idea how this helps?

Beam (recently acquired by Microsoft) does very similar to this. Rather than wait for the whole X second segment to be built and distributed, they apparently stream MPEG atoms through websockets which are them reassembled in JS and presented via media source extensions. At this point you may as well have re-invented RTMP, minus the Flash dependency :).

Yep^^

In addition there is a major encoding downside to this. If your encoder can grab a whole 2-10s chunk of video it can produce a better quality stream. This is actually _super_ important for producing good video quality (especially on things like sports streams).

In my experience live streaming events in Canada, the delay is approx 60s behind cable TV. In your estimate, is the Internet broadcaster here using extremely large segments, or is something else at play?

Personal anecdote as brazilian that watched a lot through streaming (both web and app), sometimes comparing with live digital (uhf) open TV and friends updating results through WhatsApp (as each kick result at female football shootout).

The delay would vary between 20 seconds and 60 seconds between streaming and uhf TV (just as an observation, cable TV also has a delay of about 4, 5 seconds comparing to open tv)

Would be about $93k worth of bandwidth from AWS Sao Paulo Region (about 3x more than AWS US-East)

My deepest sympathies for someone who needs bulk, reliable networking out of sa-east-1.

So maybe $9k in actual costs then.

None of these numbers add up.

If 30 million hours were watched with an average bandwidth of 2Mbps, the total data volume should be 25,749TB (or close to 26PB) and not 400TB.

Also, if the peak bandwidth was 600Gpbs, that peak could not have lasted very long. With that much bandwidth the claimed 400TB of total transferred data would be used up in roughly 90 minutes.

So either the total data transferred or the number of hours watched figure is off by a factor of at least 60x.

[ I figured "400Tb" was supposed to mean 400 terrabytes. If it's actually 400 terrabits, the numbers are off by one magnitude more ]

Wow, that's not very much at all.

In comparison, I've been using a bit over 600TB with a budget of less than $500/month. With double the budget I could easily transfer olympic amounts of data!

Really puts to perspective how incredibly accessible bandwidth pricing is these days.

Do you have a insert a dumb celebrity fan page? Or what do you do to consume that bandwidth?

Scanning.

See for example: https://github.com/robertdavidgraham/masscan

Curious how many Tb someone like twitch does per day. AFAIK they are the biggest live streaming platform, since Netflix is all HTTP.

Seems like that could be a big moneymaker for them, for these big time events that need a "live" aspect.

Akamai streamed over 250PB of Rio video over 15 days. Doubt if anyone topped that.

Twitch and Netflix use https://en.wikipedia.org/wiki/Adaptive_bitrate_streaming. So does youtube. It's all the same tech.

> since Netflix is all HTTP

You're saying you can't stream over HTTP?

Not OP, but the distinction is between live streams where the content is being generated very close to the consumption time (historically delivered using RTSP) and between static files sitting on a server/CDN (ie. Netflix, Youtube, etc.)

This is exactly the distinction I'm talking about.

Doing it live is way harder than moving static content out to all the leaf nodes once per day and serving it to a geographically small region.

Adaptive bitrate has nothing to do with the point I'm making that, AFAIK, twitch has the most "live" eyeballs out of anyone and there are special engineering challenges there.

Well, no Twitch isn't the largest. NBC/Comcast moved several orders of magnitude more of live video for the last olympics, and those numbers went up substantially this year+.

Live presents interesting challenges yes, but in some ways it is easier than VOD from a scale perspective. In live all users are watching essentially the same content++, so it is nearly perfectly cacheable in memory. That means you don't really need a ton of cache, just a cdn with lots NIC's, ram, and CPUs. You can use mid-tier caches to "fan" out data to edge caches and progressively deload origins. You can take a look at our architecture over here https://www.bizety.com/2015/07/15/deep-dive-comcast-cdn-arch....

The really hard problems with live are actually the surrounding items. If you are placing personalized ads, and 100 million people are watching a stream, your system must support 100million transactions per second. If you have QOS data from clients, you need to support 100 million TPS. Etc, etc.

Building live packagers and segmenters is also a serious bit of work. Especially to "broadcast" standards. This requires the coordination of multiple video streams subject to byzantine failure.

I guess the hardest scale problem though is probably emergency alerts. In this situation all users will tune at almost precisely the same time to a live streaming event. 30million + users can arrive on a single (previously cold) item in < 100ms. Just opening that many tcp sockets is serious work even for large numbers of servers.

+ (sauces so I don't get in trouble, full disclosure I designed and work on the systems that do this):

http://nbcsportsgrouppressbox.com/2012/08/14/ondon-olympics-... http://www.fiercecable.com/special-report/judges-tally-nbc-s...

++ http://www.cs.umd.edu/~slee/pubs/iptv-sigmet09.pdf

I'm actually not sure what's more intriguing or impressive -- the fact that the [streamer for high-profile event] spins up so much infrastructure and machinery for a duration of 2-3 weeks and then spins it down until the next high-profile event two years later, or someone like Twitch who does it on a smaller scale, but 24/7.

Though in my mind, the illusion or appearance of 'pop-up streaming' takes the award. I wonder, how much streaming infrastructure and instrumentation and code can someone like NBC reuse between two successive olympics (two years apart)?

If its 455 TB of source video then you need around 91 5TB hard drives to store that. Although they probably have a cheaper tape storage or something.

If you wanted to buy that many on Amazon you can get external 5 tb now for $109. So that would cost about $9900. Which to me is a pretty small number considering.

No, it's 455TB streamed out to all watchers, total, at an average bitrate of 2.0 Mbps. Far less unique content.

Too bad multicast streaming never really happened.

How do you think digital cable television works today? It happens, just on closed private networks.

Is digital cable television using IP? I thought it would be something proprietary.

IP to the edge (of a CMTS) and then it's flipped to QAM.

https://en.wikipedia.org/wiki/Quadrature_amplitude_modulatio... https://en.wikipedia.org/wiki/Cable_modem_termination_system

Also systems like AT&T uverse (fiber to the home) use IP multicast.

I'd be much more interested to see how much data was transferred during streaming The International 6.

According to my data, the TI6 English stream had 16M viewer hours. Unfortunately I don't know the breakdown of what bitrate everyone was watching at, but assuming a mix of 50% Source quality and 50% High quality, that would equate to about 25,145 TB if my math is correct.

Note that there's a mismatch between the listed title and the content of the article. The article refers to "400Pb (400000Tb)", not "455Tb".

This may be why there's such confusion about the numbers in the comments.

I don't watch TV so I didn't see any of it. Was it free to stream?

And then, nobody can watch because the IOC files a copyright complaint.

Didn't seem too big to me..It is like 32Tb/4TB per day. What is the number of Twitch's daily traffic, I'd say it is comparable.

Let's say that an average video is 500MB nowadays. Be it a show, a short movie, a long HD music video on youtube, or a sport event on the olympics.

Let's say that 1000 thousand people are watching the Olympic games. 500 GB * 1000 people = That's 500 GB of traffic.

Let's convert GB to Gb (multiply by 8) and we're talking 400 Tb.

So.. The title of this article means there are about 1000 people who watched ONLY one event of the olympics and then put off their TV.

Well, I thought Olympics were a big worldwide event... I was wrong.

---

Alternative: We can take it the other way around and say that there were 1000 sport events during the Olympic, and one guy to watch them all :D

Just for Fun: I'd love to see the numbers from Dailymotion.com, see how much the Olympic Games are unpopular in comparison.

A 2Mbps profile (what they mentioned) is about 900Megabytes/hour. They use ABR (hls) so, most likely there _average_ bitrate is actually only 1.5Mbps. That means about 675 MB per hour. Which means about 600k hours where consumed by this relatively small Brazilian streaming service.

Your math is wayyyyyy off.

500GB = 4Tb, not 400Tb, so it would be 100,000 people watching for a few hours. Still not much.

However, I think they miscalculated. They also mention 30M hours of video watched at an average bit rate of 2Mbps. By my calculation that would make it (30 hours is around 100k seconds, so 30M hours is about 100G seconds) 200,000Tb. That's "more than 400Tb", yes, but also 500 times as much (check: that brings it up to 50M "watcher events", quite close to the 30M hours.)

Also, if I understand this correctly, this is for Brazil only, not worldwide ("Before starting, I would like to clarify that Globo.com only had rights for streaming the content to Brazil.")

Given that and the fact that large parts of Brazil's population are poor, the 30M hours do not seem obviously wrong to me.

You missed/added a times 1000 somewhere in your math.

Keep in mind that ISPs and large streaming services each take steps to reduce the bandwidth of streaming media via different compression methods. The average video that you download in high quality may be around 500MB, but if you were to stream the same video on Netflix in a good area you would probably only consume 100-400MB of bandwidth.

500GB * 8bit = 4Tb; not 400Tb

s/500 GB * 1000 people/500 MB * 1000 people/

agreed, this really isn't that much bandwidth

2.0Mbps is an SD profile, which is worth mentioning.

Reply on news.ycombinator.com