FIFA 2014 World Cup live stream architecture | Hacker News Reader

FIFA 2014 World Cup live stream architecture

leandromoreira.com.br

258 pointsdreampeppers9929 comments

Interesting post! The author mentions that the first step was to "receive this input and to generate HLS output to a known folder". I have a couple questions about this step:

1: Did they re-encode the input? Or just repackage it?

2: I'm unfamiliar with EvoStream, but if it is ingesting RTMP and outputting HLS, why did Globo need to bother generating an HLS manifest? Couldn't they just use the one EvoStream created?

I am glad you liked. I am not the author of the blog post, but I was a member of the team.

1: We just repackage it. 2: The problem is that EvoStream stores the manifests and chunks locally and we needed high availability. That's why we use an external data storage. We have had up to 30 simultaneous streams, with 7 bitrates each and 2 hours of DVR.

Just adding one extra information, building the playlist on our side enabled us to define the DVR window (how many time one can seek back on the video). Since we built both the server and client side, it was also possible to add tags to the stream and we did it. tl;dr; It was possible to control the adaptive streaming on the server side.

When our CDN was very crowded, one tag on the HLS playlist was able to direct users to a lower quality, preventing all the users to fight for the same bandwidth and avoiding rebuffering events.

Honestly - one of the most informative, well written, extremely well referenced articles I've read literally in years. This article will teach me more than I've learned in a long time. Thank you very much for your contributions..

Just like everyone else, I really liked the article. And of course I have some questions.

What is the benefit of EvoStream over nginx-rtmp? I use nginx-rtmp as well and it creates HLS files just fine.

Also, how did you handle the network load? How many edge servers were actually delivering the content to users?

Did you use 10G ethernets, link bonding, or did you use cloud services (eg cloudfront) to deliver the content?

Thanks a lot.

* We initially picked EMS because of its API, but we're moving to nginx-rtmp.

* We used 20G ethernet, we could get 19G from each machine.

* All the content is hosted by us.

Thank you for your answer. Appreciate it and best wishes.

My favorite type of article, thanks! Even better when the stuff they're using is open source so I can check it out for myself.

It is rewarding working at globo.com with such skilled engineers and great solutions, in almost every area. Congrats to Leandro, Juarez, and everybody involved.

Thanks for sharing this! How long was your team given to complete this project?

  Cassandra response time was increasing with load to a certain point where clients started to timeout and the video playback completely stopped.

and

  After these changes, we were able to achieve a latency in the order of 10ms for our 99% percentile.

How did you test this? Are there any benchmarking/load testing tools for cases such as streaming?

If I'm not wrong, we start to access that around 1y but I think we really spent 7-10m on it.

You can send metrics from Cassandra to graphite http://www.datastax.com/dev/blog/pluggable-metrics-reporting...

Since the streaming is similar to HTTP page flow, it's not that hard.

Ex: http://blazemeter.com/blog/how-load-test-http-live-media-str...

It feels like they reinvented the wheel on a lot of things (especially on the monitoring side). I also wish there was some more detail about the caching/endpoint side of things. We rely heavily on Akamai's services for their edge servers, caching, and site acceleration. I don't see anything like that here, but maybe they're just not mentioning it?

We only implemented the dashboard (getting data from all kinds of source) on monitoring side, we relied on: logtash, graphite, elasticsearch...

The architecture up from what was described is almost purely to provide caching capabilities, aka bunch of edge servers caching content to final users.

Seems like quite a complex solution for 500,000 concurrent users.

Why not just use Akamai for RTMP ingest and delivery? From my experience they can do all that was described and at scale.

Please keep in mind that FIFA14 was one event, we still broadcast many other events and channels simultaneously ( private and public).

About why chose HLS over RTMP, in our experiments HLS showed to be easier to scale than RTMP (maybe because the latter is stateful + the current players don't have an optimized adaptive bitrate algorithm + it's easier to scale http over rtmp)

Yes agree, but Akamai can also transcode and repackage to HLS. We've used this at SportsLabs[1] to stream tens of thousands of sports events.

[1] http://sportslabs.com/

+ iOS devices won't play RTMP.

NIH syndrome.

I disagree (maybe with a little context you will do understand):

  * We were only allowed to broadcast to Brazil.

  * We already have all the servers and needed bandwidth, why would you pay when you already have all the servers?

Working in the TV industry myself, the uninevitable question on DRM springs to mind. Was these streams encrypted as well somewhere along the described process?

A great example of use the powerful and free tools. Congratulations, great article.

Great experience, thanks for sharing.

Wonderful job, guys. Will you broadcast Olympics? What will change for this one?

Yes, for sure. :) Probably, there will be some changes but we're going to discovery some of them ongoing.

beautiful! congratulations for sharing this to the community.

Really well written article! Learned a lot, also happy that it came from Brazil :)

Here's all you need to know about this company (Globo): https://www.youtube.com/watch?v=77TKLQ1op34

This documentary is still banned in Brazil and can't be broadcast offline anywhere.

This documentary is very old and pre-dates the internet.

There's no denying the extent of Globo's influence. But, compared to the 80's, it's merely a shadow of it's former self. Probably because of the internet. Also, there are other communication vehicles fighting for eyeballs now.

Politics aside, its IT arm 'globo.com' is pretty modern and open.

> There's no denying the extent of Globo's influence. But, compared to the 80's, it's merely a shadow of it's former self.

The ire lingers, though. Sometimes Globo shows up to report on something and they essentially get kicked out of the area (by the people near the event). This may be a combination of how it tends to report on certain subjects and leftover distrust from the past decades.

Reply on news.ycombinator.com