Why don't companies use FreeBSD as much in production as Linux?
As die-hard FreeBSD user and admin I'm interested to know why FreeBSD hasn't been used more often (yeah I know about Netflix and Yahoo but still)?
- FreeBSD lost the desktop battle to Linux. All joking aside about Linux on the desktop, if you want a *nix environment (and not Mac), Linux distributions are just a lot easier to set up and have traditionally enjoyed better packaging of proprietary software (graphics card drivers, RAID card drivers, applications, etc.). While FreeBSD has some very compelling advantages on the server, for those in my circle, it's not not so much better as to justify the cognitive overhead in switching between two similar, but different, OSes.
- FreeBSD wasn't a first class OS on EC2 for years. This allowed an entire ecosystem of devops tools to evolve with essentially Linux-only support.
That said, I never saw FreeBSD as anything other than a Server OS. So I wouldn't say the "desktop" comparison really ever fit in.
If it worked.
If your hardware didn't fit inside a fairly narrow box (e.g. Nvidia for graphics), things failed horribly.
I did try some of the FreeBSD desktop variants over the years (DragonFly and PC-BSD). But then you're still not quite running FreeBSD. I haven't kept up, but it looks like DragonFly is its own distinct BSD flavor now.
The sound was really good for the time if you had a supported card, real hardware mixing, /dev/dsp (or was it /dev/audio) that multiple processes could write to and it was seamlessly mixed. Using the commercial version of OSS, as I recall. In Linux at that time you either had the open source fork of OSS (which wasn't nearly as good), or raw ALSA, which was promising but buggy.
I think that right there was the problem.
Heck even Apple dont use FreeBSD with their Servers
Admittedly I haven't tried running FBSD since 4/5 timeframe and I'm sure quite a bit has changed since then, but I used to spend hours trying to patch and compile upgrades to software. The ports system that they used at the time (and might still use for all I know) was great for installing stuff, but if something didn't exist in ports or was a few version behind, you were compiling and installing from source.
aptitude has 99.99% of everything I've ever needed to install and manage and some of the newer package management systems like npm and the like cover everything else. What used to be hours each week of administration work has become minutes a month. It might be less secure, it might be slightly less robust, but overall I don't even think about systems administration anymore and (to me) that's such a huge savings in time that it's just not worth trying to go back.
And as emersonrsantos mentioned, I can almost guarantee that every hosting provider has Ubuntu in some form or another. If I need to move to a new host, I can have the system set up and be in business in probably less than an hour. What used to take a few full-time sysadmins has now been replaced by a handful of devops guys that not only manage the systems, but also help my engineers automate and improve our development workflow and pipelines. And, as he also mentioned, I can throw a rock and hit a Linux guy. Finding a FreeBSD expert is nearly impossible and they always cost a lot more.
But, it was my first unix and I've got a lot of love in my heart for the BSD way of doing things. (NetBSD and OpenBSD are other fond loves of mine). I ran a hosting company in the early 00s that was powered entirely by OpenBSD off cheap commodity hardware. And NetBSD runs on everything, which is also a pretty cool plus.
These days, I'm content knowing that my kickass Mac desktops owe their existence to the OS I grew up loving.
That hurts to read. Debian (upon which Ubuntu is based) has been around for decades (literally: it's 22 years old), and has been providing an extremely stable server, for free, with just a few apt-get commands.
> What used to be hours each week of administration work has become minutes a month. It might be less secure, it might be slightly less robust, but overall I don't even think about systems administration anymore and (to me) that's such a huge savings in time that it's just not worth trying to go back.
Or you could switch to Debian and get the same ease-of-use with better security.
If you really wanted security, of course, you'd choose OpenBSD.
Honest question: In what ways is the security story on debian better than the security on ubuntu?
All that said, I've used Ubuntu on and off since it launched, and they've been a great benefit to both Debian and Linux in general. There were some issues in the start, with not handling cooperating and resource sharing very well, there were a few sore people as a result of that -- but as far as I can tell most of those growing pains are well behind us now.
I personally sneer a bit reflexively when people mention Ubuntu on the server, but frankly that's an issue with me, and not with Ubuntu. I got burned by Ubuntu doing stuff like changing gid/uid numbers for system groups/users and little incompatibilities that made it hard to maintain services across Debian old-stable, stable and a couple of Ubuntu LTS releases at the same time for a while (the typical incompatibilities which generally have plagued Linux for as long as there have been more than one distro).
My impression is that since Ubuntu 12.04 pretty much all the major kinks got worked out, including avoiding issues with LTS -> LTS release upgrades - and Ubuntu is now as solid an OS as pretty much anything else.
As for the security bit, I think Ubuntu still comes with more out of the box, both on the server and on the desktop than Debian does. Like eg. the ssh daemon, maybe some bonjour networking stuff. Ever since OpenBSD forced everyone to re-examine what "runs by default" I prefer a cleanly installed OS to not listen to neither UDP or TCP at all out of the box. In that sense, Debian might possibly be considered "more secure" -- but I don't have the impression that there's a big difference in patch rates etc between the two.
Ubuntu/Canonical has gone to great lengths to make Ubuntu for Desktop and Server work out-of-the-box for the great majority of users -- and I honestly think they've done a great job with it. I still prefer Debian, but I also accept that it's a matter of preference not some strict criteria of superiority or the like.
Oh, and I have a special place in my heart for Debian/kFreeBSD even if I've yet to actually get to play with it: https://wiki.debian.org/Debian_GNU/kFreeBSD
Phew. Apologies for the long post - hopefully it has a few interesting nuggets.
See danieldk's comment: https://news.ycombinator.com/reply?id=12201051&goto=threads%...
I'd add that Ubuntu encourages the use of PPAs, which are a security nightmare.
But apt-get has undergone many improvements because of the swell in Ubuntu users. And that user-base has brought more documentation and energy to linux than any other distro.
Why? Ubuntu is based on Debian and I guessing it shares and contributes to same package ecosystem.
> Debian (upon which Ubuntu is based) has been around for decades
Found Debian, after being around for decades, never quite won the desktop market. Debian got there perhaps 80% ofthe way, but the final 20% was just hard to achieve, and I think Ubuntu put a great effort into that 20%, polishing it up and making it a much better experience. After trying Debian, CentOS, Mandriva, Gentoo for my dev machine, I ended up with Ubuntu, and with it I stopped worrying about my OS. Almost everything works. I don't even think about or notice the OS anymore. And that's a good thing!
So, back to Debian, once I want to deploy something and I've played with it on my Ubuntu dev machine, which OS would I evaluate first? Ubuntu Server, of course. So I think the server story paradoxically often starts on the Desktop.
Maybe, but Ubuntu Server has serious downsides when it comes to security updates. Canonical only releases security updates for main/restricted, while Debian releases security updates for the whole distribution. So, if you are using packages from universe/multiverse (which I guess a lot of people do), security updates are not guaranteed. Or as Canonical puts it:
Canonical does not provide a guarantee of regular security updates for software in the universe component, but will provide these where they are made available by the community.
In the early 2000's, I had to set up a dedicated CVS server for my 10-person startup. I was the dual hat "dev & admin" guy. We were small, but had good desktop machines with good SCSI disks in them (top of the line Seagate, if I remember right).
I set up both a Linux and a FreeBSD dedicated CVS server. We were all happy to try one server for a day, copy the repo to the other one for a day, and try things out.
Well, FreeBSD was a bitch to set up compared to Linux and was easily 10X slower in every way measurable. Like, "cvs up" would take 5 seconds from the Linux server compared to a minute on FreeBSD (yes, same repo...). Hopping on to the FreeBSD box locally showed that every kind of disk activity was way slower than Linux.
I went to the FreeBSD newsgroups and got laughed at. Not a single piece of helpful info. I did get several large words thrown at me about how I didn't understand the benchmarks and the performance shouldn't be noticeable to the end users. At least one guy took to emailing me directly about how I shouldn't be comparing Linux to FreeBSD.
After 2 or 3 days of wrestling, I powered off the FreeBSD box, installed Linux on it, and never considered FreeBSD again.
Politics/marketing:
* Most "unix" admins only know linux and will advocate for it vigorously because it is so much better than.. "what do you use again? Fedora? Ah, FreeBSD, something with F, I knew it!"
* Management not always listens to reason but just wants a discussion go away, so they listen to the loudest advocates.
Knowledge/Community:
* It is easier to copy a solution to a linux problem from stack overflow than to read the FreeBSD handbook, understand the problem and fix it.
* FreeBSD only is fun when one has a higher level of expertise: Knowledge of sh and at least a basic understanding of make are required, being able to at least read c code makes life much easier.
* Reading docs is hard. FreeBSD forces one to understand the standard unix tools that come with it. That means one has to spend some time reading the docs (or at least skimming over them so one knows where to look when the need arises). If one does not understand the tools, even simple init scripts are black magic.
* No exposure to FreeBSD at all: most hosting companies won't even list FreeBSD as an option (though in most you'll find some unix geek who'll happily connect KVM or IPMI and insert the install-CD for you).
* The FreeBSD community is much less forgiving than the linux community: Ask a question that can be answered easily by reading the handbook or some man-page and the response will probably be silence (rarely flaming, just silence).
And finally also some points where one could argue that it makes sense not to use FreeBSD from an economic point of view:
* When using FreeBSD one is regularly forced to clean up linux-BS when venturing outside what /usr/ports provides: #!/bin/bash - or even worse #!/bin/sh that actually wants a bash - is one of the most common problems.
* When compiling software from source that does not come from /usr/ports one regularly has to do research what $leenox-distro-package XY provides because documentation just gives command lines for the most common linux distributions. "Soo.... what exactly does that software need to compile when the docs tell me to simply apt-get foo-23.5 and bar-42.666?"
Some of my datacenter clients use freeBSD for various bits, and I have been that guy. In the end they're migrating away because it's incredibly hard to find experienced engineers. What you describe as "reading docs is hard" can be equated to "my team will be slower for negleglible gain." Dicking around with ports and make is fun, but at the end of the day, we seek lower latency services and faster outage response times.
EDIT: there's also a network effect in puppet / chef / ansible -- More work for your operations team when every community module for managing services doesn't support your platform of choice.
> What you describe as "reading docs is hard" can be equated to "my team will be slower for negleglible gain."
Yes, but only from a very short-sighted point of view. I cannot count the "devops"-meetings I had to attend as a consultant during which I hacked a command line that solved the problem the meeting was supposed to make a plan for and estimate costs... I dare to argue that letting the ops learn the ropes on company time would have been much cheaper than my fees plus costs for working time employees spent on that meeting. But I understand this is very hard to quantify and that in a startup culture people don't want to think beyond the point where the financing is used up.
My fav. test for sysop/devop candidates: "Tell me how large the home directory of all users who use [t]csh as a login shell is on that machine; no, you cannot install anything, there is no perl, ruby or python. You have 90 seconds.". 99% fail. I've even interviewed people who applied for a sysop/devop positions who could not set up a host if not through puppet because they know sh1t about the target OS.
PS: Thanks for being "that guy" who allowed me to use a mature OS. People like you allowed me to have a 336 day uptime as the lower limit. Most of my machines have more than a thousand days of uptime :)
> My fav. test for sysop/devop candidates
But how do you query your LDAP server without installing additional tools? Because in 2016, we have more than one machine. Hundreds. Grepping /etc/passwd is missing the forest for the trees.
You can have all the shell experts, I need people who can write Chef code that passes peer review. kitchen lets us poke around the OS all we want to inspect proper functioning.
> People like you allowed me to have a 336 day uptime as the lower limit. Most of my machines have more than a thousand days of uptime :)
AFAIK, none of the BSDs have online kernel replacement facilities. So you willingly admit you haven't upgraded your kernels in years? I know fBSD has a reputation, but every once in a while this still happens: https://threatpost.com/freebsd-patches-kernel-panic-vulnerab....
You are missing my point, it's about being able to filter and transform textual output. If a sysop can only do what the UI provides s/he is useless when creative solutions are asked for.
> You can have all the shell experts, I need people who can write Chef code that passes peer review. kitchen lets us poke around the OS all we want to inspect proper functioning.
Every good sysop I've met can write your Chef code. Understanding system basics and managing with high level tools is not mutually exclusive. Tools like Chef are the result of exactly those sysops automating what they could. You know that saying? "Tomorrow I'll replace you with a shell script." ;)
> So you willingly admit you haven't upgraded your kernels in years?
Yes, I only update when a security problem concerns me and that is pretty rare with custom kernels that only have what is required. Even my desktop setups need a new kernel only once a year or so.
I'm (a little) ashamed to admit tech support via trolling. Linux sucks because you can't even [whatever].
It is nice to get a quick response of "it's just [whatever], noob".
Although, that was a last resort. I've met a couple of people that reread man pages every year, i've never quite picked up that practice. Once you get the hang of parsing the super terse examples, man is pretty great.
Indeed, it is. But one won't learn anything of value. One just learns the solution to a very specific problem. If one chooses to learn the basics, more often than not one will decide that it is quicker to dive into the problem and fix it than to "troll" tech support.
I mostly agree with you, but there are some unique facts that, back in the day, you just sort of had to know. something like
echo 1 > /proc/sys/net/ipv4/ip_forward
Recently I found my only machine running FreeBSD (as a testing machine) was kernel crashing occasionally. On a whim I googled '<machine name> kernel crash', and found a page which discussed an extra linux boot-time flag I had to add, to disable some feature of the graphics card. I couldn't figure out how to do the same thing on FreeBSD, so just switched to Debian, and added the magic option. No more crashes. Now the machine runs a FreeBSD VM on top of linux.
This is my favorite part of the Linux community. Ask something that you aren't sure of, and if it's wrong, you'll get quite some volume of responses correcting you!
Personally, I've been running FreeBSD (including some commercial installs -- years ago) for about 16-17 years.
Linux (especially Debian-based distros), is usually the first UNIX flavour that people coming from Windows get in touch with. While FreeBSD has excellent documentation, Ubuntu and others simply have the larger newbie-friendly community.
So far, all the guides I've seen for dual-booting have been for Linux, and they've all been straightforward.
Netflix I "think / read" is only using FreeBSD with their Storage Appliance. So that is relatively small parts in number of Servers. Most of their operation, those Chaos Monkey killed machines are all EC2 and Linux.
Whatsapp is Erlang and FreeBSD, a real rare bleed. But i read Facebook is hiring Engineers to make Linux Network Stack better then FreeBSD. Bold claim on the Job Description, then there were rumours Whatsapp moving to integrate with Facebook, i.e moving off FreeBSD to Linux. I doubt thats an coincidence.
Despite the many / some similarities between FreeBSD and OSX, even Apple aren't using any FreeBSD on their Servers. At least there hasn't been any evidence in hiring.
So really, which large, Internet Company is actually using FreeBSD?
However in 1992 and 1993 there were some internal political problems in BSD land (which eventually led to FreeBSD spinning off) and the "USL v. Regents of the University of California" lawsuit also cast a shadow on the legality of the BSDs.
During those two years Linux kept developing, jumped ahead and basically grabbed all the mind-share and marketing share.
The BSDs have been playing catchup since. For instance with fewer developers they had problems supporting a wide range hardware so were not even an option for many.
- It's easier and cheaper to find Linux sysadmins than BSD ones
- There's more commercial software supported on Linux
- There's more companies offering support for Linux
- Linux is more portable than BSD
- Linux is one of the standard server OS offered by providers, BSD isn't
I'm surprised that more embedded devices don't run some sort of BSD because they wouldn't have to release code. The PS4's OS is FreeBSD, which they did release a lot of code in return, but they don't have to open source or release all their code (i.e. DRM and other "Security" based stuff) if they don't want to.
Linux has had a ridiculous amount of work from a variety of sources in it's drivers. I once used a git animation tool (forget the name of it, sorry) on the linux kernel source, and the area where the drivers were put was buzzing like a beehive, as people from all sorts of companies were making sure their stuff was supported on linux.
Probably Gource I would imagine http://gource.io/
Linux runs on all these platforms and more - that toaster only had an ARM board sticked to it
And that is a 2-way road, where some people cry on forums and maillists anyway, that companies tend to use BSD and not return to community. There is no need to paint it as a win-win situation(distribute proprietary derivate software) when it clearly has its problems too.
that's the big one there. My favorite, Hetzner (not only mine, they have a ginormous amount of dedis) offers BSD but a) not all servers are compatible and for older servers you can't even know whether they are without booting into Linux first b) it's much more of a hassle like installing Linux gives you a ready-to-go system while the BSD install needs tweaking with sysctl and network both.
From what I gather from all these replies people's main concern is that FreeBSD is generally harder to use/configure and management is hard to persuade into supporting it because it would make DevOps less productive.
From the standpoint of a person who run FreeBSD in production on both bare metal and VM's and manages several commercial applications and websites, I must say that plenty (if not all) of these phobias are unjustified. Yes, there are some edge cases but 95% of the time, there's nothing new to learn you already don't know, except new package manager commands and slightly different file system layout.
Not using awesome technologies like ZFS, Boot Environments, Jails, Qjail, Poudriere and awesome PF is just a plain missing out in my opinion.
I personally don't care about desktop, there are already 2 awesome OS's that majority of people use for a reason and that's pretty much enough I think.
I personally heard nothing but praises from people who are long term Linux users/admins who tried it.
Once people figured out couple of OS specifics its all breeze :)
Maybe I'm sheltered.
But Linux has come a long way since then. The major issues have been resolved and IMO the overwhelmingly larger ecosystem Linux enjoys outweighs most if not all advantages FreeBSD might still have on a technical level.
About a year ago I wanted to replace newlines in a file with "sed" on FreeBSD. I couldn't.
At first I thought there was an error on my side. I was very surprised to find out that you could not do that at all, it said so right there in "man sed".
Dirty hacks to work around this problem posted on stackoverflow did not work on that machine either.
Tried to mess with newlines with python. Turned out there was some kind of other problem if you open files in 'wb' mode under FreeBSD.
This experience with some of the most basic tools not working as they do on GNU/Linux made me want to avoid FreeBSD if possible.
Those people who are familiar with a Unix variant are more familiar with Linux. Linux is different enough from *BSD that what people do learn about Linux doesn't translate.
Linux entry level is actually easier than Windows these days. In fact when I look for candidates if they only have Ubuntu / Docker on their resume its trash.
What used to be considered entry level skill sets are now called "Michael Jordan" level skillsets ( kernel compile / tuning , C debugging etc etc )
Currently I use Linux because it makes me money. I use the BSD's because I like them
I hope there's some wiggle room in there for folks who may know more but cater their resumes to what they want to do and/or the buzzwords recruiters are filtering by.
I still run FreeBSD 8.4 on one production system, and recently decommissioned a FreeBSD 5 box that had been happily humming away in a colo for 10 years. I learned FreeBSD before Linux and prefer almost everything else about it...except that part about updating software.
If you take security seriously, you apply security updates in a timely fashion. For a while I was diligent about it on my FreeBSD boxes, monitored vuxml, and updated vulnerable packages regularly with this:
portaudit -a |
sed -ne 's/^Affected package: //p' |
sort -u |
xargs portupgrade -P -rv
Near as I can tell Ubuntu/Debian wins here because it freezes packages alongside the OS release, except for backported security patches if you're on LTS. FreeBSD has only one ports tree. It's in constant flux, and (in my experience) is constantly broken. Why can't ports be branched off alongside the OS release and receive security backports? Maybe FreeBSD doesn't have the manpower to do this, maybe it's cultural, I'm not really sure. What I can say is it's relatively easy to check out, tweak, build and install ports on a case-by-case basis if for some reason you need the latest and greatest of something. I don't see the value in constantly having the latest and greatest of everything though, and it even seems a little antithetical to FreeBSD to me.
Anyway, so port upgrades suck, but base upgrades also suck. Doing an `apt-get dist-upgrade` to go from Ubuntu 12.04 to 14.04 "just worked." Rebuilding world to upgrade from FreeBSD 7.x to 8.x worked, but just barely, and the whole process scared the shit out of me. Random incompatibilities continued to crop up for some time after.
I think this one comes down to the integrated base userland + kernel in FreeBSD versus the "everything's a package" approach in Ubuntu/Debian. Kernel upgrades are much more common on Linux. Not that this is an objectively good thing, but rare events don't tend to get tested and optimized like routine ones do.
Note: if you're a diehard FreeBSD user and you've figured out how to keep your system up-to-date with minimum fuss, please school me.
Accompanies with ZFS boot environments there's literally minimal fuss about upgrading and worrying about possible breakages.
ZFS really is amazing. Before I ever actually tried it, I couldn't understand why people were so amazed by a filesystem. It's incredibly hard to "go back" and I often feel a bit "crippled" when managing machines not using ZFS.
People both get more value from Linux and add more value to it by using it and writing software for it. Take docker or nix for example, huge value for Linux and not so much for systems, where they don't work or aren't first class citizens.