Hacker News Reader

Comment by brendangregg | Hacker News Reader

This is great news, and we're already using ZFS in production on Ubuntu in a few areas at Netflix (not widespread yet).

Ubuntu 16.04 Tracing ZFS TIME COMM 06:31:28 dd 06:31:34 dd 06:31:35 dd 06:31:35 dd 06:31:35 dd [...] # ./zfsdist Tracing ZFS ^C operation = 'read' usecs 0 -> 1 2 -> 3 4 -> 7 8 -> 15 16 -> 31 32 -> 63 64 -> 127 128 -> 255 256 -> 511 512 -> 1023 1024 -> 2047 2048 -> 4095 4096 -> 8191 [...] The current also comes with enhanced BPF, the new Linux tracing & programming framework that is builtin to the kernel, and is a huge leap forward for Linux tracing. Eg, we can start using tools like these: https://github.com/iovisor/bcc#tracing

View on HN

What about FreeBSD?

Does that imply Netflix is transitioning away from FreeBSD? If so, why?

Netflix cloud: AWS EC2, tens of thousands of cloud instances, mostly Ubuntu.

Netflix CDN (Open Connect Appliance): lots of physical boxes, FreeBSD.

Any reason why you don't use FreeBSD everywhere (or Ubuntu everywhere).

Why two different OS'es?

It's really two questions: Why choose Ubuntu for the cloud, and, why choose FreeBSD for the CDN. We believe that's the best choice for both environments. I was trying to type in an explanation here, but that's really something that will take a lot to explain (maybe a Netflix tech blog post).

That would be a great post.

If you do write that blog post, it would be cool if you not only covered the FreeBSD vs. Ubuntu aspects of the choice, but also the Ubuntu vs. other Linux aspects (particularly Debian).

How does BPF compare to dtrace?

If you browse some of the _example.txt files in https://github.com/iovisor/bcc/tree/master/tools , you'll see it's solving the same problems we used to solve with DTrace, plus a few extra. Here's a couple of the ZFS examples (since we're talking ZFS):

  # ./zfsslower operations slower than 10 ms PID    T BYTES   OFF_KB   LAT(ms) FILENAME 25570  W 131072  38784     303.92 data1 25686  W 131072  38784     388.28 data1 25686  W 131072  78720     519.66 data1 25686  W 131072  116992    405.94 data1 25686  W 131072  153600    433.52 data1 operation latency... Hit Ctrl-C to end. : count     distribution : 0        |                                        | : 0        |                                        | : 4479     |****************************************| : 1028     |*********                               | : 14       |                                        | : 1        |                                        | : 2        |                                        | : 6        |                                        | : 1        |                                        | : 1256     |***********                             | : 9        |                                        | : 1        |                                        | : 2        |                                        | BPF interface we're using (bcc) is Python for the frontend, and C for the backend. It's currently much more verbose than DTrace, and involves writing 10x the lines of code. For some immediate use cases at Netflix, that's not a big problem, as staff will be using BPF via a GUI (Vector), not writing these tools directly.There's also high level features it's still missing (like tracepoints and sampling), so what will be in Ubuntu 14.04 won't do everything, but it will do a fair amount: most of those _example.txt's. Some use a newer BPF interface (linux 4.5), and we've been putting the legacy versions in an /old directory specifically for Ubuntu 16.04 users.

Does the current publicly released version of Vector support BPF? Or is there perhaps a PMDA that allows BPF support?I'm following along with all of this pretty excitedly, and crossing my fingers for a Linux tracing book with BPF, ftrace, perf, etc. to read through and keep on my shelf next to your performance and dtrace books ;)

Vector doesn't have BPF support yet. When it does we should be open sourcing it. As for a book, I'll try to get it done. :)

So good to see ZOL has come so far!

Reply on news.ycombinator.com