Hacker News Reader

Comment by drewg123 | Hacker News Reader

Speaking as somebody who works on the FreeBSD kernel at Netflix, the whole article is kind of nonsensical. We don't use FreeBSD because of "latency". We serve clients over https that could be tens or hundreds of ms away using the kernel TCP stack. The folks who care about latency are high frequency traders, who care about every nanosecond. They tend to use userspace / hardware offloaded solutions and do things like busy-waiting for messages.

View on HN

Drew

Have you ever consider using Dragonfly. It appears to dramatically beat in perf test FreeBSD (2017).

https://leaf.dragonflybsd.org/~sephe/perf_cmp.pdf

I’m curious to hear your take on Dragonfly since you’re a kernel developer.

Thanks in advance for all your work.

Edit: more recent links (2018) with even higher perf

https://leaf.dragonflybsd.org/~sephe/dfly1.pdf

https://leaf.dragonflybsd.org/~sephe/dfly1_p.pdf

I have looked at Dragonfly with interest, due to how they manage VM. I really like how they shard the page queues to be per-cpu. With that said, a lot of work has been done by jeffr in the last 6 months to a year to improve NUMA performance (which we partially sponsored) which has dramatically improved VM scalability even on single socket boxes, and which has allowed us to remove a lot of our local hacks.

I considered giving Dragonfly a try in the past, but just never had the time. There would be a depressing amount of work before we could even consider Dragonfly, mostly centered around our async sendfile (which is now upstream in FreeBSD), our TCP changes: BBR (not yet upstream), RACK (now upstream) and TCP pacing (now upstream). Not to mention unmapped mbufs (not yet upstream) and kernel TLS (also not yet upstream). Also, the last time I looked, Dragonfly did not have drivers for Chelsio or Mellanox 10/25/40/100GbE NICs. Without even just one of these things, performance will be so bad that any comparison testing would be meaningless.

Thanks for the thoughtful reply.

Just a thought, the Linux/BSD community might hugely benefit if someone with your deep knowledge released a set of perf test scripts that anyone can run locally to regression test network perf. That way, the OSS community can integrate those perf test scripts into their commit/regression test pipeline.

It would be great to have this kind of test, but often the regression test is going to be run the system on production load. Benchmarks have a way of testing something, but not quite what you need to test. The interactions between the whole system are important.

For the Netflix boxes, they're pushing 40gbps+, not a lot of the community is going to be to be able to test that, unless they have fairly expensive networks laying around.

You're spot on. We don't really have good benchmarks that approximate our production traffic.

BTW, 40Gb/s was so 2015. I have a box serving at 156Gb/s :)

As someone who grew up around Telex machines and current loop interfaces: consider me impressed!

Oh mine! I record there was an article about pushing 100Gb/s not long ago and hitting memory bandwidth limit. Now it is 150Gb/s already ? Are you guys going to try something crazy like 400Gb/s ? At this rate Netflix could start selling their Appliance as another business.

> BTW, 40Gb/s was so 2015. I have a box serving at 156Gb/s :)

That's right, I wanted to put 100+, but I wasn't totally sure. I stopped counting when you got way beyond the 20G connectivity on the servers I manage.

For what it's worth: Sepherosa Ziehau spent some years serving out huge streams of video in a company similar to Netflix, in China. He used DragonFly. So, it may not have the same changes you are listing, but it may be closer than you think.

If I may ask, why do you use FreeBSD in particular then?

Native (properly working) ZFS, I don't need any other reason :)

It works on linux, but to be able too boot from it, still a huge hassle.

I tried to intall Ubuntu into root ZFS and it wasn't much of a hassle, just simple howto, everything worked.

I've been running root on ZFS (GELI- or LUKS-encrypted, no less) since FreeBSD 7 and Ubuntu 12, and to be frank the story on Linux just isn't that great. Documentation for root on ZFS only exists for Arch, Debian, Gentoo, and Ubuntu, with only Debian and Ubuntu having documentation for encrypted root ZFS pools. Last I checked, not a single Linux distribution's installer supported ZFS, so if you want root on ZFS, you must install Linux by hand. That's great if you have unlimited time, but it sucks for any kind of deployment at scale.

Contrast with FreeBSD, with first-class support for installing root on ZFS since 10.0-RELEASE, four years ago.

I’m going go out on a limb abd speculate there might be different hurdles for a single installation vs many installations.

Does it have he same level of QA?

Reply on news.ycombinator.com