Linux Profiling at Netflix | Hacker News Reader

Linux Profiling at Netflix

slideshare.net

134 points2510c39011c532 comments

Can anyone speak as to why Netflix needs so much computing power? As far as I can tell, content delivery is handled separately, so all these talks are about the non-content side.

Sure we need to see various personalised lists when first connecting, but no activity while actually watching things. The personalisation seems very light - eg their lists of comedies look like they show the same to everyone. And it doesn't seem intelligent at all, like for example hiding items you have already seen. https://news.ycombinator.com/item?id=9083196

Because they have thousand Linux nodes for Cassandra.

The microservice architecture results in a larger footprint than you might find with a more monolithic app stack; which provides us with improved engineering velocity and reliability. Although the content playback is served by our CDN, all other device interactions are handled by the services on AWS. We also have some blogposts which provide a glimpse into some of the work we do behind the scenes (http://techblog.netflix.com/2015/01/netflixs-viewing-data-ho...). Oh yeah, we also encode and run our big data services on AWS...

This still doesn't answer why it is so large. Or more accurately my experience as a Netflix customer doesn't reflect the apparent size of all these backends. Compare to something like Reddit where every page is customised, I go through lots of them, and my interactions are frequent. My Netflix interactions look like recently viewed being custom, and everything else being generic (same for all users). If you told me my recommendations are recalculated once a month I'd believe you. The majority of time is spent viewing which is just periodic progress updates. In other words my user experience suggests Reddit having an infrastructure many times larger than Netflix.

I read that Netflix encodes each video 100+ times to ensure good performance across a wide range of devices. In addition to the "social network" side of the site (recommendations, what your friends watch, etc.) there is a lot of processing going on in the background.

I vaguely recall the number being encoded 120 different ways. In any event this is independent of the number of users, and proportional to the amount of new incoming content. It also isn't real time, and doesn't really matter how long it takes other than marketing deadlines when they announce a new season of something far in advance.

The recommendations don't show any evidence of being calculated frequently, nor do they have to be done in real time. There is certainly way less appearing to go on than a site like Reddit.

"There is a lot of processing going on in the background" doesn't really answer the question :-)

Sure, this processing does not need to happen in real time, but it does need to happen.

My point was that to assume the user-visible portion of the site accounts for most of their computing needs is probably incorrect. Video encoding was just one of the first things I thought of, and I bet there are many more.

I admit "processing going on in the background" was a real hand-wavey way to phrase it, but I think you know what I mean. It's an iceberg - there's a lot happening under the surface. :-)

I don't work on the recommendations team but I feel I can safely say your recommendations are calc'd each day. Unfortunately I can't share more about our recommendation model other than to say its a strong business advantage.

Still sidestepping what the heck all the machines are used for :-) You seem to be hinting that the majority is the recommendations, but even that shouldn't require such large scales. It also doesn't have to be that timely (eg it won't matter that much if the recalculations were once every 23 hours or once every 22). As a user it would be nice if this "strong business advantage" was useful.

"but no activity while actually watching things" is a false assumption. About 6-7% of upstream traffic at night in the US is Netflix...

The SCALE 13x videos are going up here: https://www.youtube.com/channel/UCN2nbMPLJWv3Y__4JuF_hMQ Does anyone know in what room and day Brendan spoke?

https://www.youtube.com/watch?v=dhDpncTDDvI#t=892

You can format the time like this:

https://www.youtube.com/watch?v=dhDpncTDDvI#t=14m52s

While it's great to have all the talks captured like this (and I get to go see some other talks I missed), the quality is a bit fuzzy. If that's annoying you too much, there was a second camera in the room. I'll tweet the video when it's posted (@brendangregg).

We're in the process of editing all the videos down to individual session recordings. We're all volunteer, so apologies for the delays. If anyone out there would like we're happy to get extra hands with the editing work.

For those who missed today's post https://news.ycombinator.com/item?id=9121015 because it got downvoted (wrongly) for being a duplicate: accompanying blog post and video now available: http://www.brendangregg.com/blog/2015-02-27/linux-profiling-...

I really like the workflow chart on slide 22. I can't help but think this is a useful way to visualize common workflow sequences of commands (think git add, git commit, git push).

Honestly I wouldn't mind if man pages came with ascii art to visualize this type of documentation.

It often feels like the only users of perf are its developers and Brendan Gregg.

I use it - perf is wonderful. Back in the dark days of grad school, getting performance counter data for your program required hacking your code to call into PAPI (http://icl.cs.utk.edu/papi/overview/). Being able to tell a program, "Hey, run this for me, and tell me how many branch mispredictions there are. And where they happen" is like magic.

If you do any performance analysis of software, learn perf.

Official manual: https://perf.wiki.kernel.org/index.php/Main_Page

Brendan Gregg's perf page: http://www.brendangregg.com/perf.html

I beg to disagree. For instance, it's been running for years sampling Google's fleet [1] (the paper mentions oprofile, but that has been replaced quite a while ago).

[1] http://courses.cs.washington.edu/courses/cse590g/11sp/36575....

I used it to debug some performance issues in a graphics engine we use at my company. It helped me understand what the low-hanging fruit was.

This talk was absolutely fantastic at SCALE13x. Lot of good content that will help a lot of people get started with CPU profiling.

Those flame graphs look hella useful, hopefully the JDK patches make their way in so we don't have to futz around with customizing too much. Thanks for this!

Absolutely, I had no idea there is a way to export the symbols and make them known to perf. Its magic, and I want it.

Omitting the frame pointer is a default on x86_64 its specified in the abi so it's going to be mighty tough to change that. I don't understand why you recommend not using the debug info when you already have to use it for symbol name resolution.

Are you talking about http://www.x86-64.org/documentation/abi.pdf, and footnote 7 on page 16, which reads:

"7 The conventional use of %rbp as a frame pointer for the stack frame may be avoided by using %rsp (the stack pointer) to index into the stack frame. This technique saves two instructions in the prologue and epilogue and makes one additional general-purpose register (%rbp) available."

%rbp is mentioned elsewhere, eg, "optionally used as frame pointer"

This ABI does not appear to specify that the frame pointer is omitted. It uses the words "may" and "optionally". Which means, the gcc engineers can choose to do what they want. And they should choose to keep the frame pointer _by default_, and _optionally_ omit it (for when the sacrifice is deemed worth it): -fomit-frame-pointer. This would better follow the language used in the ABI! :-)

We've found debuginfo to be large (size, bytes), which can slow down instance creation, which means we must run with a higher headroom to accommodate. A stripped-down debuginfo that just enabled stack walking might do.

Compress the debug info? Gdb handles it no idea about perf though, there is also dwo to remove redundant debug info.

Why FreeBSD for content delivery?

From https://openconnect.itp.netflix.com/software/index.html:

For the operating system, we use FreeBSD version 10. This was selected for its balance of stability and features, a strong development community and staff expertise. All code improvements, feature additions, and bug fixes are contributed directly back to the open source community via the FreeBSD committers on our team. We also strive to stay at the front of the FreeBSD development process, allowing us to have a tight feedback loop with other community and partner developers. The result has been a positive open source ecosystem that lowers our development costs and multiplies the effectiveness of our efforts..

So some aren't confused. Content delivery is on FreeBSD, not Linux.

ZFS drives?

Surely they use Linux, yet they still can't offer a proper Linux client. DRMed much.

Reply on news.ycombinator.com