Netflix End of Series 1 | Hacker News Reader

Netflix End of Series 1

brendangregg.com

I've been following Brendan since the Sun Microsystems days with his Dtrace book which opened my eyes to the world of system call tracing from user-space to kernel. Linux at the time had a hodgepodge of tracing utilities (SystemTap, Ktap etc..) that usually required kernel modules to work properly. Strace was my goto in Linux back then but slowed down the target binary a lot. Thank the universe for eBPF and Brendan's current books that today we have powerful tracing capabilities in Linux that rivals Dtrace in FreeBSD/Illumos. If you want to understand the internals of an operating system and how things work, without being a kernel developer, systems performance engineering is a way to go.

Netflix has generally amazing latency, I can feel that the engineers work really hard on fixing all latency issues, except when it totally fails: offline mode.

I always have some films downloaded on my iPad Pro that I want to watch when I don't have internet (travelling on long plane flight), and with Netflix always the best films that I want to watch are ,,expired''. It would take 1 second to ,,renew'' them, but I have to wait until the end of the flight to have internet access again, and then I forget it, as I don't look at downloads anymore. It's really frustrating that I see that I have something downloaded, taking space on my SSD, but can't be used when I need it.

I wish it could just renew the video certificates automatically whenever I access the internet, and generally it would be great to have better analysis tools for offline / crappy internet for companies.

Usually I use Youtube more on the plane, because it doesn't play these games with me.

I have the same experience. It feels like the download experience was only really made for the “I’m watching a series and want to keep watching it on the plane” use case, and not the “I downloaded 10 episodes of Micky Mouse for my 3yo on our last trip a year ago and now she wants to watch them again but I can’t cause…” use case.

something that was solved with recordable media, then with the internet instantly via usenet, then pirate sites.

these companies will never learn, they literally invent their own problems as soon as they think they can force their bullshit on us.

Arrrrrr if only there was an easy solution me matey

Congrats to Brendan and he’s been huge to continuing the advancement of FreeBSD.

I can only hope that he brings FreeBSD to his next employer & is allowed to continue his contributions.

What does he do in the BSD space? I always regarded him as Linux tracing/performance god.

Before he got his Netflix special, he worked for Sun then Joyent in Solaris/BSD/SmartOS in stuff like ZFS and DTrace.

The Linux stuff was a money-grab, some say he sold out.. :)

I've been clear on HN multiple times that I saw no future in the Solaris/SmartOS ecosystem by 2014, which is one (but not the only) reason I left it. I was right too: https://www.itjobswatch.co.uk/jobs/uk/solaris.do

Ignore the troll and thank you for all your content Brendan!

Relax it was a joke. You can state your appreciation without disparaging others who didn't follow your direct means of communication.

I thought it was pretty clearly a joke. Maybe a ";)" emoji would have been more explicit.

Ah, it _was_ a joke. I really wasn't sure.

>The Linux stuff was a money-grab

That seems like an odd conclusion to me. If you've got deep expertise in the spaces he does, and the server market moves mostly to Linux, getting involved there seems unavoidable. Lingering on the dying SmartOS ecosystem, even if it was technically better, wouldn't have served anyone well.

Edit: Ah, as mentioned below, perhaps a joke? I'd attributed the smiley face only to the latter part of the sentence.

I think it was a joke.

The odd conclusion to me is how the last line I've post was interpreted as a serious statement.

Didn't the "Before he got his Netflix special.." or the fact Linux is pretty much the only game in town with no sign of even being seriously challenged served as a hint?

Context is hard with just text. Smarter people than me were confused :)

I thought the :) was obvious. So I wouldn't worry about it too much.

Although at one point I thought Brendon not only abandoned solaris but also BSD.

You're joking of course, but it was me who wrote that, and I stand by what I wrote.

I touch it now and then; search for "BSD" on https://www.brendangregg.com.

Netflix has hired more great BSD folk, so I haven't needed to do a lot on it for a while. I do enjoy it though, and it's usually much easier to read and understand the kernel code.

Most recently I've been talking to folk about eBPF on BSD. Maybe I should do a blog post about it (force me to go talk to everyone again and get their updates).

Thank you Brendan. I work in systems performance engineering in finance and always have been your humble and fanatic follower.

I learned a lot from your books, posts and talks and use your tools (mostly bpftrace and flame graphs for the JVM) on daily basis. I can't thank you enough for the wealth of wisdom you shared with us, mortals!

Can I ask how you got into such a role? Is it at a major bank or some kind of consultancy?

I started as a java developer around 20 years ago; in those times it was called enterprise java and at some point I joined a consultancy that had many clients in telco industry. That meant there was a lot of opportunities to get exposure to performance troubleshooting. At first it was only on the JVM level but you quickly realize the virtual machine doesn't run in vacuum an the OS as well as HW are equally important layers of software execution stack.

I devoted a lot of time to learning about systems engineering and modern hardware. When switching jobs it was always with the purpose of getting more and more exposure to performance-focused software engineering.

In my case the last 12 years was was exclusively big banks and one exchange (LMAX, UBS, Goldman Sachs, Morgan Stanley, HSBC). From what I can tell there's more opportunities to get your hands dirty with perf on the buy side (hedge funds, prop trading shops) but I always got lucky with the big market makers and got involved in amazingly interesting things anyway.

I absolutely love this line of work and wouldn't trade it for anything else but it sometimes costs you some grey hair especially if you're dealing with the very bleeding edge of hardware and kernel/OS level bits (think spending weeks chasing misbehaving icache on a heavily overclocked cpu).

This job is very rewarding but requires real passion and constant learning.

Hmm, do you think eBPF will have less overhead than dtrace?

Brendan has my dream job.

I even hoped joining Netflix in the future to say hi to him :)

Doing performance analysis feels like the final thing to fully understand the whole stack and I always was curios on how things work.

Let's see were he will go. Perhaps it's easier to follow but I doubt it.

There are companies I'm following for a while because certain positions are just seldom pop up.

When I joined sap I heard the term headcount for the first time but makes it more understandable.

Peformance (beyond analysis) is one of the most interesting things to do beyond building things. It's needed and relevant in many settings and you can (have to) do it without the JD (and IMO you should at least initially). You'll find plenty of this at any relevant size infrastructure place (there are a lot of them).

The principle to do whatever you love is to "ask for forgivness not permission". Feel free to understand why you're solving what you're solving in order to challenge it and do it right.

I'm currently at a very big company (not faang) and I had the chance to do a little bit of performance analysis.

But unfortunate it was a totally unoptimized thing so only quick low hanging fruits.

Like io bound but they didn't though that component is the issue as it was not CPU bound, bad SQL queries and unoptimized caches.

Nothing big or difficult to analyse :-(

I learned about one person who is only doing java Performance analysis and he said it took a while to be known in the industry good enough that he can do it fulltime.

And he has to travel to allot of customers.

Hey it’s fine to build a name for yourself while knocking out low hanging fruit. It’s still bringing useful and meaningful results in a professional setting.

I work on performance and I enjoy it even more than writing features, honestly. There’s much more freedom in it in a professional setting because you generally get to define the subtasks under the goal of “make it faster/more efficient” whereas with a feature you’re usually told what specific thing you’re supposed to make, and your flexibility is usually just picking how to implement that. And the problems are almost always challenging and require technical (and occasionally, political) insight to fix. With features you might ship something that’s loved or a dud, but improving the performance of something is basically always a good thing.

Well, he moved to Australia last year.

https://www.brendangregg.com/blog/2021-05-29/moving-to-austr...

I’ve built and run a performance monitoring platform/team at a large tech company. I’ve also worked in security for the past few years.

Both have similar problems: if the business goals don’t align with reducing latency or improved security posture then your detailed analysis will get parked in a ticket queue and promptly forgotten. Your OKRs will be missed. You will also be fighting the tide against new features developed sloppily- as latency will increase and then investment in your team seems like it’s not working to the executives. Fortunately we had a supportive directive but being siloed into an infra org didn’t give us the juice to influence product roadmaps.

We built a lab for testing against devices, did a bunch of work to isolate noise, integrated automated regression testing into CI to combat this. It still didn’t move the needle frankly, as the teams building products were not held accountable for the regressions (and don’t forget that it’s likely the right move for the greater business health!).

Netflix is unique in that latency is very core to their product.

I guess at that point of ones career they can basically write their own job description and budget, and people will line up to pay them whatever that description says.

This reminds me of this post titled "Netflix is a truly great company": https://www.teamblind.com/post/netflix-is-a-truly-great-comp.... BTW, their compensation system is the most fair in the industry.

Thanks for sharing this. Why do more companies not follow this philosophy?

Perhaps one advantage for Netflix is that they’re largely a one-trick pony. They don’t have 20+ products fighting for attention and engineers within the company.

My take is that that philosophy needs a system, or culture if you will, to support it. Not every company has such supporting culture.

Off topic: I’m a bit surprised about Gregg’s desk (pre-pandemic). I imagine he’s getting a top level salary at Netflix but yet he’s got a small desk in what it looks to me a shared small office (or perhaps is that a mini open space office? Can’t tell).

A number of times people have asked about my desk over the years, and I'm curious as to why! I've visited other tech companies in the bay area, and the desks I see (including for 7-figure salary engineers) are the same as everyone else, in open office layouts. At Netflix it's been open office desks, and all engineers have the same desk.

Does some companies give bigger desks for certain staff, or offices, or is it a country thing (Europe?).

I think given your visibility and profile (compared to many tech workers), the previous poster expected a grand office or vast desk, certainly different from the setup “everyone else” has.

I understand why people might think that, but I'm not sure it'd be great for the culture for me to be different from my coworkers. We're all working on important things, and we all need to collaborate together.

Absolutely, I agree with you there. I’m relatively junior in my team and my team is quite absent of clout within our organisation as we’re a new team.

Our correspondent teams are much more approachable when it comes to collaborating than I’ve experienced in the past - it’s something I find very refreshing.

Looks pretty similar to the desk I had at Amazon UK. You have a nicer chair though.

Almost every US company I've been at gave at least 2 stand-alone monitors to every engineer. That's what stuck out to me.

Ah right, people have asked about that before. I'm just happy with a single normal-sized monitor.

Maybe related, but we once went around the Netflix parking lot with Blippar to look at the KBB value of the cars, and found that I was driving the cheapest car at Netflix (at least until we got sick of looking). Mine was a 2001 Toyota Prius, KBB value ~$1800. Not the cheapest car to ever park at Netflix, just the cheapest on the day!

I don't find the desk itself surprising, but I am surprised by the open office plan. I've found sharing an open space with more than two or three other people extremely distracting. I'm sure it depends a lot on the personalities of the coworkers.

Why is open office plan surprising? Its practically de rigueur at tech companies.

Is there a story behind those cricket stumps? ^_^

Yes, I helped organize the Netflix cricket practice sessions. Netflix has a cricket team (in engineering) that in the past has played in the GPL: Google Premiership League, along with several other tech companies. Lots of fun!

We practiced on baseball diamonds, and played league games on the Google soccer field.

When I ran the BPF tutorial at USENIX LISA 2019, I had one finger in a splint from breaking it playing cricket. :-)

This reply made my day ^_^

Cheers and thanks for everything.

Used to work at Sun in Ire and i had a massive corner desk cubicle, whiteboard, 3 monitors etc. Then working in banking it was pretty big cubicle areas (NYC/London) now reduced down to more normal desk sizes.

I'm more shocked at the modesty of your monitor

I don't know about larger companies, but I work for a much smaller company and every person has an office with a door. After having my own space, I'm not sure I could go back to an open office plan again.

I haven't had my own office. That feels like such a 1950s Mad Men type of thing. Has to be pretty rare outside of accountants and lawyers today.

But I will say the absolutely best I've had was 3 devs to a single office, with desks oriented so that the monitors face the walls for privacy. We could chat throughout the day, but you could also get real work done. I've done the open floor plan thing. Never again. There is a certain constant level of anxiety with having people floating around you and talking nearby all day long.

> Has to be pretty rare outside of accountants and lawyers today.

Add WFH to that list!

I like how you have your own books on the desk :) ultimate flex!

Speculation:

One of the game companies that do distribution (valve, epic)

Working with old colleagues (as a C-level?) at oxide.

One of the major cloud providers, very likely microsoft.

Oxide, maybe?

Perhaps since some of the top ex Joyent engineers are there also..

Yep, that’s my guess.

That was my guess too

OpenAI, on AI alignment instrumentation?

Given that he just stated in this thread he doesn’t believe in Solaris/SmartOS ecosystem - probably not then.

https://news.ycombinator.com/item?id=31052035

Brendan is truly one of the best educators in tech engineering. He may not think of himself as such, and he's also obviously a fantastic engineer, but what really stands out to me is his ability to communicate ideas coherently.

So much of the "staff+" engineer world is hot air, and Brendan has always struck a contrast with his straight forward, information-dense blog posts and books, and down-to-earth communication style. He explores, and often defines, the frontiers of what we do and then brings back lessons.

I'm a huge fan, and can't wait to see what he does (and shares) next.

First time I’m really hearing about Brendan Gregg’s work. Not that I’m someone who should know, but I think this puts eBPF on my future studies list. Plus Australia sounds very cool. Currently working towards RHCE certification myself after a long time of swearing I don’t need it. (I want a new, more advanced job so I need it.)

I am so grateful somebody’s using FreeBSD in 2022 in production. I do wonder whether there’s anything that could get it more use. Maybe a certification and backing company like red hat does for Linux. Or something more harebrained like a smartphone app geared towards systems management for the upcoming generation. I’ve heard they’re generally less interested in this stuff.

I say this because it’s awesome to see Netflix having people so deep in the tech stack. But somehow still feels a little fragile.

His systems performance book is also ridiculously pragmatic and eye opening and definitely worth a read.

This video is how a lot of people first learned about Brendan: https://www.youtube.com/watch?v=tDacjrSCeq4

I’m really curious to see how the rock star of cloud computing follows up Netflix …

Congrats on getting out of the asylum Brendan

I was happy to see a Thinkpad docking station on your desk, good choice on laptop!

I've followed you awesome work on Linux performance tuning over the years. All the best for whatever you're doing next and I hope it ends up in another book for us mere mortals to read :-).

I hope he will be contributing to the performance space as much as he's been at Netflix. His books and blogs have helped many including me better understand the whole stack.

Good luck in your next adventure and a big thank you for all your past work, blogs, tools and books. Much appreciated and helpful in my work.

I like Brendan's Commodore 64 mouse pad in the pic! I have the same one! :)

What brand for notebook lock on the left table ?

When engineers like Brendan Gregg complain about American immigration and visa processes, you know something is really really f-ed up in how America treats humans.

Reply on news.ycombinator.com