Hacker News Reader

Hehe, I can feel your pain. Had a discussion just a few days ago with some guys demonstrating the typical enthusiasm about cloud stuff, especially AWS. Because after all, Netflix is using it, and they create 1/3rd of all Internet traffic, right?

There's this very widespread misconception about Netflix being a monolithic service running on Amazons' cloud infrastructure, even though the truth is that just the "rather boring" routine stuff like billing, view history tracking, suggestions, everything necessary to show the UI is running there. Netflix does a good job at this, but after all it's not that impressive when you've architected some distributed systems yourself. Not even their extreme take on the microservices concept - that is after all just a nice way of letting their devs do "their thing" in the way they want with as little restrictions from the environment as possible (which basically only works because they clearly have above-average-competence devs who can deal with these degrees of freedom).

What's really crazy is the way they squeeze unimaginable amounts of bytes per second out of modern hardware and into the internet infrastructure. They saturate 100Gbit links that usually serve an aggregation of many boxes in a datacenter with just ONE box! This is way beyond what even most above-average devs are capable of doing - you NEED old-school guys which still managed to stay on top of the crazy stack created by the evolution of hardware and low-level software in the last decade. There's not many of those out there, and Netflix apparently managed to catch a good bunch of them. These guys do the magic, and the magic they do never touches that damn Amazon Cloud. It just floats way above it.

Netflix is my goto example of "do your core competency in-house, outsource the rest".

They do one thing themselves in their stack: their distribution network for content, and they do an incredible job of it. Every post I see about Netflix's CDN for video is insightful and a learning experience.

Then they throw a CRUD app in the cloud on top of it and call it a day. Okay, a little simplified -- there's still some neat tech in the DRM, in load-balancing the CDN, and in keeping all of their tech highly available. But conceivably, Netflix could retain much of their value by simply offering all the content to other websites that displayed it to customers -- the hard part of what they do is the CDN (and contracts with content owners), and opening their platform to other interfaces doesn't change that. (Heck, Netflix might be worth more if they opened their content to other interfaces, since they're not actually very good at the front end experience.)

But when I was doing consulting (about cloud stuff), that was my advice: do the core of your business yourself (eg, CDN) then offload as much of the rest as you can.

That's what I find most impressive. The insane amount of bits they do from a single FreeBSD box.

And I concur as well on the BSD crew responsible for engineering it. The grey beards are getting harder to find as they retire and I weep each time one does.

You stated it very well, especially that last paragraph.

I'm busily bringing lmbench up to date and seeing if I can get my mojo back enough to hang with these guys and do some work. It's both exciting to think about kernel work and depressing to realize so few people understand it any more.

Is it every ISP, or only certain ISPs who opt in?

I remember OpenConnect being mentioned back when the ISP I use came on board. The service has been really good. I haven't gotten a single buffer in like 3 years.

If anyone is curious, glance through this. It's a pretty cool initiative:

https://openconnect.netflix.com/en/

If you get crappy Netflix service you should call your ISP and point them to this page.

> I wish hacker news got excited about the filling the pipe post and less excited about this thread.

That's disingenuous considering the number of votes and comments each thread have received. HN IS more excited by the pipe post as reflected by user participation.

Would love to hear more detail insofar as this article is concerned.

The 100Gb pipe article was off the front page and not really commented on that much. I see post after post about containers this and vagrant that, all sorts of posts about stuff that I find pretty uninteresting. Something as remarkable as the 100Gbit thing gets a little attention but it is clear that it's not that interesting to the majority of the people here.

I'm just depressed that real low level systems stuff seems to be not sexy any more. When I was at Sun the kernel group was the unchallenged top of the heap, it was the place to be. When I was fixing source management at Sun (as a hobby, it wasn't my official job) they asked me to go work in the tools group. Are you frigging kidding me? Noone in their right mind would leave the kernel group for the tools group. Sorry to be snooty but that just wasn't a thing.

I just got back from a FreeBSD conference at Netflix and was asking about the state of the world and it's depressing. People don't seem to write solid papers like they used to. Sun produced papers on vnodes, the VM system architecture and implementation, Sparc, I wrote one on making UFS perform like an extent based file system. Who is doing that now? I looked in the usual conference proceedings and it was full of academic stuff, not very interesting stuff, but no industry stuff.

I think that must have been common. Nobody good worked in tools. I managed a couple sunos and later solaris boxes. Userspace was a disaster. I ended up replacing it all with GNU and every piece I installed had a warning. Don't use Solaris make. Don't use solaris tar. Don't use the solaris compiler, etc. Ended up having to install gcc from binaries, then build zip, make, etc. Even tail (max 10k lines) and awk (max 16 columns or something) had issues. While debugging even xterm would die if you made it too wide.

Once you replaced the userspace with GNU + X11 the kernel was solid, although SunOS and early solaris didn't multitask well. For some workloads I'd end up disabling all but one CPU.

So I believe you on the Solaris userspace, it was full of System 5 garbage. GNU was a much better answer.

But on SunOS 4.x, you had userspace problems? That was a pretty stock BSD userspace with a lot of bugfixes. Every open source makefile in the world just worked when you typed make on SunOS. Seemed pretty solid to me, I think the only thing I would do is add perl, it took Sun a long time to decide to include perl (if they ever did).

Is there a GNU/Linux project equivalent of the netflix/freebsd sendfile? Besides this use case, what is your assessment of the linux (netfilter) vs bsd netstacks? What kernel is going to be at the center of faster and faster switch fabrics?

Senfile is a syscall in the Linux kernel since 2003~~

I wrote up a hand wavy way of doing a generalized sendfile here:

http://mcvoy.com/lm/papers/splice.pdf

and linux took some of that. As I recall, they don't use it much because the user land apps (like web servers) end up touching the data for TLS (https).

Netflix/FreeBSD are working with the NIC folks to get the byte by byte TLS pushed down in the card along with the TCP offload. I think they'll get there first because BSD is so heavily used by CDN people (not just Netflix, Limelight is very active in BSD as well).

Can you recommend a more technical version of OP's article or a video-taped talk?