Comment by TheCondor | Hacker News Reader

Comment by TheCondor | Hacker News Reader

Parent Full thread

The community is such that if one of either FreeBSD or Linux really outperformed the other in some capacity, they'd rally and get the gap closed pretty quickly. There are benchmarks that flatter both but they tend to converge and be pretty close.

Netflix has a team of, I think it was 10 from the PDF, engineers working on this platform. A substantial number of them are FreeBSD contributors with relationships to it. It's a very special team. That's the difference maker here. If it was a team of former Red Hat guys, I'm sure they'd be on Linux. If it was a team of old Solaris guys, I wouldn't be surprised if they were on one of the Solaris offsprings. Then, Netflix knew that at their scale and to make it work, they had to build this stuff out. That was something they figured out pretty quickly, they found the right team and then built around them. It's far more sophisticated than "we replaced RedHat Enterprise with FreeBSD and reaped immediate performance advantages." At their scale, I don't think there is an off the shelf solution.

I think you're close but missing an element. FreeBSD is a centrally designed and implemented OS from kernel to libc to userland. The entire OS is in the source tree, with updated and correct documentation, with coherent design.

Any systems level team will prefer this form of tightly integrated solution to the systems layer problem if they are responsible for the highly built out specialized distributed application we call netflix. The reasons for design choices going all the way back to freebsd 1 are available on the mailing list in a central place. Everything is there.

Trying to maintain your own linux distro is insanely difficult, infact google still uses a 2.2 kernel with all custom back ported updates of the last 30 years.

The resources to match the relatively small freebsd design and implementation team are minuscule compared to the infinite crawl of linuxes, a team of 10 freebsd sysprogs is basically the same amount of people responsible for designing and building the entire thing.

It comes down to the sources of truth. In the world of fbsd that's the single fbsd repo and the single mailing list. For a linux, that could be thousands of sources of truth across hundreds of platforms.

> infact google still uses a 2.2 kernel with all custom back ported updates of the last 30 years.

Linux 2.2 is from 1999. It can barely do SMP. That's pretty much a crazy person claim.

Googlers say ProdKernel is merged forward every few years:

> Every two years or so, those patches are rebased onto a newer kernel version, which provides a number of challenges.

https://lwn.net/Articles/871195/

> Every ~2 years we rebase all these patches over a ~2 year codebase delta

https://events.linuxfoundation.org/wp-content/uploads/2022/0... (and many other places)

I understand the point you're trying to make and I agree that FreeBSD tends to have cleaner code and better documentation at different levels but I don't think that it makes it that much more difficult. If you dropped in from a different world and had zero experience then I think you're right and a systems team would almost always pick BSD. Any actual experience pretty quickly swings it the other way though; there are also companies dedicated to helping you fill in those gaps on the Linux side.

I've built a couple embedded projects on Linux, when you're deep on a hard problem, the mailing lists and various people are nice, but the "source of truth" is your logic analyzer and you debug the damn thing. Or your hardware vendor might have some clues as they know more of the bugs in their hardware.

Fair points taken, i was a bit zealous in my use of the word any, the word many is more correctly applicable.

In regard to sources of truth, i mean from the design considerations point of view. For instance, why does the scheduler behave a certain way? We can analyze the logic and determine the truth of the codes behavior but determining the reason for the selection of that design for implementation is far more difficult.

These days yes, off the shelf linux will do just fine at massively scaling an application. When netflix started building blockbuster was still a huge scary thing to be feared and respected. Linux was still a hobby project if you didn't fork out 70% of a commercial unix contract over to rhel.

The team came in with the expectation and understanding they would be making massive changes for their application that may never be merged upstream. The chances of getting a merge in are higher if the upstream is a smaller centralized team. You can also just ask the person who was in charge of the, let's say for example, the init system design and implementation. Or oh, that scheduler! Why does it deprioritize x and y over z, is there more to why this was chosen than what is on the mailing list?

The pros go on and on and the cons are difficult to imagine, unless you make a vacuum and compare 2024 linux to 2004 linux.

> Any systems level team will prefer this form of tightly integrated solution to the systems layer problem if they are responsible for the highly built out specialized distributed application we call netflix

If any team would prefer this, then as gp asked: why is Linux overrepresented at other Non-Netflix content-delivery shops?

> infact google still uses a 2.2 kernel with all custom back ported updates of the last 30 years.

Say what?

They are trying to get off or have gotten off their kernel fork called "Prodkernel" for some time now.

https://lwn.net/Articles/871195/ https://events.linuxfoundation.org/wp-content/uploads/2022/0...

And ProdKernel generally lags mainstream by only a few years, as can be seen in the things you linked to.

The change being talked here is moving from merge ~2 years to merge all the time. Saying they're stuck on something from 1999 is ridiculous.

Practically this just doesn't matter all that much. You can prefer one approach to the other and that's all fine, but from a "serious engineering" perspective it just doesn't really matter.

> Trying to maintain your own linux distro is insanely difficult, infact google still uses a 2.2 kernel with all custom back ported updates of the last 30 years.

2.2? Colour me skeptical on that.

But it's really not that hard to make a Linux distro. I know, because I did it, and tons of other people have. It's little more than "get kernel + fuck about with Linux booting madness + bunch of userland stuff". The same applies to FreeBSD by the way, and according to the PDF "Netflix has an internal “distro”".

The problems Google has is because they maintain extensive patchsets, not because they built their own distro. They would have the same problems with FreeBSD, or any other complex piece of software.

Reply on news.ycombinator.com