The FreeBSD/Firecracker Platform
daemonology.net
daemonology.net
I'm pretty sure that discoveries like these is why it's good to have operating system diversity. Conversely, sometimes I do wonder if having everyone on the same OS would allow us to have way more cool things and still have less bugs. But it doesn't matter too much because it'll never happen...
I'm not saying you're wrong or right, but it's worth remembering that it's hard to guess how it'll be in the future. Maybe Apple acquires Microsoft at one point (or vice-versa) and eventually ends up buying Linux (somehow, don't ask me how), and we'll end up with one OS for absolutely everything. Weirder stuff have happened.
There used to be multiple contenders for what would end up as "the internet", with many countries having their own versions. I'm sure at that point some people had similar sentiments running through their heads, something like "wow, wouldn't it be cool if all these various networks was just one instead? But that'll never happen..."
Similarly how we today look at messaging services not being interoperable with each other, but seems this is slowly (veeery slowly) changing.
> One of the best features of FreeBSD is that it can be used as a general-purpose operating system. That means that FreeBSD can be used as a server, as an embedded system, or in networking.
Same for Linux.
> FreeBSD provides simplified software management with the help of its packaging system and the ports collection.
Not a ports experts but the difference between ports and Gentoo portage does not really seem that big, certainly nothing of FreeBSD itself makes it more suitable to working with ports than a Linux kernel. And there is guix and nix also which really should cover most software management needs and flexibility anyone has.
> One of the many great advantages of FreeBSD is that it allows you to customize the operating system, according to your needs.
Again Gentoo, nix and guix covers this quite well.
> You don’t need lots of cores – but you can definitely run it on huge multicore systems, can easily run a highly graphical interface, or choose the simpler way; FreeBSD will support you either way.
Linux definitely does scale up and down, maybe FreeBSD scales down a bit smaller than Linux, but I would have to see numbers to back this up, and I don't think it is that significant in difference.
> FreeBSD has been ported to a variety of instruction set architectures.
Again this is not something unique to FreeBSD and Linux has a proven track record of portability.
> FreeBSD includes two native file systems, the Unix File System (UFS) and the Z File System (ZFS).
I guess "native" ZFS is the closest it comes to a UVP, but given Linux support for ZFS is quite good to the point where ZFS on root is possible I'm nto sure it qualifies.
Not saying FreeBSD is not great, but not sure it really has any unique value propositions.
As far as I know gentoo portage is based on freebsd's ports. Where Gentoo has different kernels (linux or freebsd) and an ever changing userland, FreeBSD has consistency they make a complete operating system they don't bolt stuff together like Linux distros do. The holistic approach of freebsd results in a more cohesive and considered environment linux doesn't have this.
The difference is not that one has the command pkg and the other portage but how the entire environment has been considered to work with ports.
> Linux definitely does scale up and down, maybe FreeBSD scales down a bit smaller than Linux
In my experience they both scale the same, it makes sense that they would both watch each other and implement any improvements they seen since they are both open source.
> I guess "native" ZFS is the closest it comes to a UVP
If you are looking for a killer feature that destroys linux you aren't going to find it they are both very similar, its like looking for a UVP for Fedora vs Gentoo vs Debian they all do pretty much the same thing but in different ways. I use Fedora on desktop and Debian on servers as they both seem to be well suited to those environments, I could just change one to work like the other but why bother? Same is true of FreeBSD.
Kernel and Userspace are developed, and packaged together, in the same repo.
Lack of such benchmarking, will be indicator of "no one cares on that platform anymore".
Nginx can be other one "indicator" - Nginx started as FreeBSD focused product in company heavily focused on FreeBSD (I know couple of persons from there by that times), nowdays not even mentioning ARM64 in list of supported product
> FreeBSD 3 — 12 / i386; FreeBSD 5 — 12 / amd64; FreeBSD 11 / ppc; FreeBSD 12 / ppc64;
from [1] and [2]. Not a big deal for _now_ but, but - more and more ARM servers in major hostings, even Hetzner has 2 of them. Seems this train have been missed too.
Let's see will I win my bet or not.
[1] https://nginx.org/en/#tested_os_and_platforms [2] https://docs.nginx.com/nginx/technical-specs/?_ga=2.23463900...
The same way as in freebsd "service blabla start" it's a compatibility hook.
Fedora (and its derivatives like RHEL/CentOS/etc) still has the "service" command, it forwards to the corresponding systemctl command. The same for Debian (the comments in Debian's "service" command even mention it's a modified version of the RHEL/Fedora one).
My fist Linux was from 2003-4, Debian Woody. Why I learned was until 2007-8 mainly the bunch of tools from Debconf (the installer, update-alternatives, apt, dpkg-reconfigure, make-kpkg (to compile a newer kernel), debhelper and so on), not the Unix way to do things. In 2006-2009 I knew how to compile Mplayer from source (tgz) and a bunch of other tools (Cedega from CVS) with apt-get build-dep to fetch the deps and a bit more, that's it.
From 2009-2010, I've got a brief match with Slackware and Arch, and from that, OpenBSD. OpenBSD was dumb simple in design, seriously. With Linux you had to do voodoo with Artsd/ESD disabling them and then more fuckery with UCM files and DMIX. OSS under BSD's (I liked FreesBIE, a lovely live CD with FreeBSD and XFCE) just worked. And it ran faster than Linux.
Also, the Bash guides from the Howto's were utterly difficult, much more than the Ksh ones from Orelly (learning Perl for scripting was far easier, trust me).
Everything is in one place, and it is maintained. In contrast, Linux requires you to go look up documentation in a lot of different places, and it is highly dependent on distribution what to do in a given situation.
One reason for this is that FreeBSD maintains not only a kernel, but also a userland. This means a change in the kernel can immediately be followed by appropriate changes in the userland and vice versa.
But my experience with FreeBSD's docs are even better - and most of that experience is quite old at this point. I can't imagine how much better the situation has gotten since FreeBSD only seems to mature and not degrade (based on comments like yours which are very common).
I also seriously dislike the BSD's tendency towards making --help useless, forcing you to bring up the man page. This is just for practical reasons. It's easier to hit up arrow, dash dash help pipe less than up arrow, ctrl+a, man, alt+f, ctrl+k to kill the arguments you've already written :)
/me adds a new project to the backlog...
Man pages in OpenBSD for instance are a "must". In the GNU world, often man pages are really badly maintained pointing to GNU info. Also, the so called "Linux Howtos" were a disaster full of obsolete points.
Just compare the OpenBSDxy.html upgrading page (where m and n are release numbers) for OpenBSD releases (and the FAQ) to the Arch Wiki.
OpenBSD also strikes a very good balance between "informative" and "terse" that should be held up as a model for others. Unfamiliar things may need a few readings to get everything, but it's all there without being a toilet-clogger.
With NetBSD and OpenBSD everything looked sane. They explained you everything, from /etc/profile, managing services without making you mad with runlevels, to launching X and so on.
twitch
Small English correction: the noun directly after the verb "to explain" must be the thing that is being explained. What you wrote means "they explained [anthk] [to everything]". They gave an explanation of you to the whole universe?
This is a common mistake but it winds me up every time.
"They explained" + the thing they explained + optionally to whom or what they explained it.
It should be: "they explained everything to you."
For example, when there is a security bug in Linux you can shutdown those servers and spin up FreeBSD ones while the patch is being made.
Until around 2003 to 2005, FreeBSD had much better performance and reliability than Linux.
That changed after Intel introduced Pentium 4 with multiple threads in 2003 and AMD introduced CPUs with multiple cores in 2005. Before that, multiple hardware threads were available only in very expensive computers, which were inaccessible for most people.
Linux was adapted very quickly to work on multi-core or multi-thread CPUs, while FreeBSD required many years for that transition.
At that time, Linux got a great advance in performance over FreeBSD, which was difficult to recover later.
Nowadays, FreeBSD has device driver support for a much less number of devices than Linux, so you must be more careful when choosing hardware for it.
Nevertheless, even if I use Linux in laptops, desktops and computational servers, I have continued to use FreeBSD in networking servers until today.
The main reason is that the administration of a FreeBSD server requires much less time than the administration of a Linux server (which requires much less time than the administration of a Windows server).
With FreeBSD, I may not do anything manually for years, if I have configured it to make certain automatic updates.
With Linux, at least with the distributions that I like for other reasons, I have to always supervise the updates, because the updates frequently break things. (I assume that this might not happen with something like Debian stable, but whenever I have looked to a Linux distribution like that, it was too antiquated for my needs.)
One of the main culprits is always Python, because there are a huge number of packages that insist that they are compatible only with a certain Python version, and they conflict between themselves even when I have all the possible Python versions installed. I could understand that passing from Python 2 to 3 has broken things, but why some packages like Python 3.9 and dislike 3.10 is beyond me. Also the compatibility with various Qt versions frequently causes update problems.
On FreeBSD all the system is more consistent, so I need to spend no more than a couple of hours for a server after several years of 24/7 functioning, when I make some hardware update and when I might replace the major FreeBSD version at the same time.
> With Linux, at least with the distributions that I like for other reasons, I have to always supervise the updates, because the updates frequently break things
What distributions have you run into these issues with and how often is "frequently" for you?
I'm running Arch Linux on my desktop and bunch of servers, for more than five years each one and can count the number of breakages on one hand. On the other hand, I run Ubuntu on my laptop and stopped counting the amount of issues I've had with upgrades.
But I'd consider that amount of breakage from my Arch installs very infrequent, but maybe I'm comparing it to the wrong thing, as as I said before, I don't have any experience with BSD systems, maybe breakage there is even more uncommon?
from the docs:
> FreeBSD is bundled with a rich collection of system tools as part of the base system. In addition, FreeBSD provides two complementary technologies for installing third-party software: the FreeBSD Ports Collection, for installing from source, and packages, for installing from pre-built binaries.
Thus, when FreeBSD guys say "my system is solid" - means he can use ping/top/csh. But if update breaks say Firefox - it's not part of the system and haha, system is stable!
Also, I am an OpenBSD user (not FreeBSD), and I can say packages are much better maintained than in the 99% of the distros.
No anything specific I have to put against Firefox or any other application here.
Most of the programs that I use on FreeBSD come from ports, not from the base system (they are various networking applications like squid or samba or nginx), even if I use some functions from the base system, e.g. firewall/NAT/routing, some times the NFS server.
Nevertheless, I did not have problems with inconsistencies in the ports collection, and that includes Firefox (which I use some times together with vncserver, so I can use it remotely even if I do not normally install the X11 server on FreeBSD).
It is true however that on FreeBSD I have used only seldom programs with GUI interfaces, Firefox being an exception to that, and those are usually among the programs more likely to be broken by updates on Linux systems.
The only time things "broke" for me is when X11 on FreeBSD switched to libinput by default.
Never had issues with anything else:
- nvidia drivers always worked, unlike ubuntu that tried to install opensource-not-working-driver every time it gets
- My keyboard remapping worked every upgrade. Ubuntu broke it major release, Arch broke it from time to time. Fedora I didn't even bother.
- Most of my software was always on bleeding edge release when possible
- Docker in bhyve worked better than on linux because I could just restart VM
- Not once, my bootloader went foobar in 20 years (with GRUB I live in constant fear)
Now, if FreeBSD folks stopped with perfectionism and finally landed GPU and WiFi drivers...I'd be still using it. Now I won't switch back though, I'm far too friendly with Nix.
Many, many moons ago a video game called Return to Castle Wolfenstein was released, and in addition to a Linux server where you could host multi-player games, it actually had a Linux client.
I played on the Linux client for a while with an NVidia GPU. But my main desktop was FreeBSD at the time, and so instead of dual-booting I used FreeBSD's Linux API/ABI emulation to play RtCW with FreeBSD NVidia drivers. I even got a few more FPS under FreeBSD.
* https://en.wikipedia.org/wiki/Return_to_Castle_Wolfenstein
Debian and Ubuntu breackage was guaranteed. On RH/RHEL, it was not easy somtimes.
Arch had two major switchs (SystemD and anothers).
On OpenBSD, you just read the upgrade guide, download the bsd.rd, boot it and upgrade. Or with sysupgrade from a few releases. Much reliable than Linux.
agreed about the ease of openBSD upgrades from one release to another
So I most frequently use Gentoo or some distribution derived from it, e.g. Funtoo.
I have switched to Gentoo around 2002/2003, after using a few older distributions, including Slackware, Redhat and SUSE.
During the first decade of using Gentoo, the Portage collection was better maintained and I would have never had problems with upgrades, except that when I have tested KDE 4 I was astonished by how the team which hijacked its development had been able to remove all the features that I liked in KDE 3.5, so I have wiped KDE 4 and I have reverted to KDE 3.5.
Then, for a couple of years I had to fight more and more at each upgrade to avoid the breakage of the preserved KDE 3.5, until that became so difficult that I gave up and I replaced KDE with XFCE.
Then for a few years there have been no problems with upgrades, but during the last decade they have begun to appear sporadically. In many cases the upgrades still work without surprises, but every few months there is one that fails because some package does not compile for various reasons, e.g. because some library is not listed as a dependence so it has not been updated, or because there is a circular package dependence that appears only with the specific combination of USE flags that are set on my system, or because there is some conflict between the dependencies of some packages that prevent them to be installed together without certain workarounds.
What I like at Gentoo is that it permits an extreme customization of the system, but that is what makes very difficult for the package maintainers to test their package build instructions, because it is hard to generate all the combinations of options that might have been chosen by some user.
The breakages are not frequent, but because they happen sometimes I am not confident to allow them to be performed automatically, without supervision, like I allow them to be done on FreeBSD.
I would prefer a package system like that of Nix, but I have not found yet enough time to play with Nix, or maybe with Guix, to evaluate which will be the consequences of converting to it.
This is something is sincerely miss in Linux, where more often than not you have to either decide between having bleeding edge packages on a bleeding edge system (like Linux, or Debian Sid) or having to deal with old packages and third party repositories (like on Debian or Red Hat).
Also, the fact the whole system is shipped, tested and built by a closely knit group of developers is a huge bonus to me - the overall experience feels polished and well integrated, something Linux distros often seriously lack. At times on Linux it just feels as if some packages have just been bundled haphazardly together with zero care about documenting their interactions or how they are supposed to work, or that the packagers do not agree with upstream and do random modifications that just complicate things (like on Debian or Red Hat). Lots of stuff in FreeBSD has no upstream - FreeBSD and the other BSDs write and maintain their own kernel, libc, core utils, init system, bootloader, ...
Q2: a bit different, but similar as the end goal - immutable infrastructure where you say update your AMIs and migrating services there, shutting down old "unapgraded" servers at all?
Not with OpenBSD. If you run -current, please, do not mix -release and -current pkgs/ports.
NetBSD it's bound to major releases:
https://www.netbsd.org/docs/misc/index.html#package-manageme...
My upper limit would be ~ 5000 servers in use, 3-500 web related developers for that system/servers.
Jails are fantastic for segregation of services. Linux has comparable things, but I've always been more comfortable in the jail environment. It just makes more sense to me.
FreeBSD is much more of a Unix-y way of doing things, and if that's the kind of thing you prefer, it's excellent. I'm not a systemd hater in particular, but I do prefer Ye Olde Wayes.
If its tweaking, then im trying to remember any kind of tweaking of the system beyond sysctls that I'd practically need ..cannot much.
Jails probably OK, but as you mentioned it's nothing unique like it was till 2006..
I don't see your selling point.
(Another person who comes to mind is Matt Dillon of DragonflyBSD and his huge contributions)
> The FreeBSD kernel now handles such duplicate environment variables by appending suffixes, so that we end up with virtio_mmio.device, virtio_mmio.device_1, virtio_mmio.device_2, et cetera,
Was there any discussion of a cleaner alternative to this, perhaps extending the existing APIs with the ability to retrieve multiple values for one key (without breaking backwards compatibility, obviously) or concatenating the values with some sort of delimiter (colon, literal new line, ascii RS, or even null)?
Not being able to tell whether or not foo was specified twice or foo and foo_1 were specified separately just gives me flashbacks to a bygone era of osdev and gives me security creepy-crawlies.
(also how do you handle `device=xxx device=yyy device_1=zzz`?)
We end up with device=xxx and device_1=yyy but if you mean that someone specifies device_1=zzz after that, it would show up as device_1_1=zzz. I'm inclined to file that one under "if a user is deliberately obtuse, they get what they deserve".
No, I meant the first example. I'm trying to think of realistic non-obtuse cases like an entirely different spec calls for name_1, name_2, etc. but for some reason a numbered key is repeated. It's mostly an academic concern though, no worries!
Thanks for the awesome work you do for the FreeBSD community, as always.
The biggest strength is the ports / package manager works better than anything I have tried on linux and it sticks to unix principles closer than linux.
The biggest weakness for freebsd is docker and mono support is very poor. There is a desktop distro but in general any gui stuff linux beats freebsd hands down I only use freebsd for server stuff.
I use it for production web servers, backup / storage servers, routers, virtualisation servers. I find its rock solid and more consistent than linux.
Security wise there are some minor differences. For instance bhyve supports virtio-rng but firecracker doesn't want to: https://github.com/firecracker-microvm/firecracker/issues/16... .
I think if your app requires fast boot time and your app supports that than it's fine (so services that spin up and down on demand) but apps that daemonize for extended periods of time or take forever to initialize probably not a great fit.
But to answer the question: On small VMs, we can reach /sbin/init in under 50 ms and /etc/rc finishes running in under 500 ms. (Larger VMs take longer -- past a few GB of RAM, the time for initializing paging tables and launching APs starts to dominate.)
I've long tolerated Linux but never loved it, and it's moving away from anything I'm interested in with each year, it seems (the competing mostly-bad image-based app distribution systems and the way Wayland's going are... yikes). Glad FreeBSD's there as an alternative.
It turns "how do I install and run this?" from a bunch of platform- and distro-specific instructions into `docker-compose up` for everything, with the bonus that the compose file also tells you exactly where data & config for the service live, so it's also concise, guaranteed-accurate, and extremely useful documentation. Old LTS Ubuntu? Can run the same version of a service, with the same command, as on bleeding-edge Arch. On Arch, exact same command. Debian, Red Hat? Same command. MacOS, if you've got docker set up, same command. Windows, ditto. The latter two, non-natively, sure, but it works. Learn docker, and you can use it everywhere, unlike becoming proficient with, say, dpkg/apt or portage or pacman. It can even kinda save you from having to worry about how a given platform's init system and process management works, at least for simple use cases.
Again, if you're doing most or everything on FreeBSD, development and hosting both, those benefits mostly vanish. But the reality is many workplaces or even personal dev+hosting environments are mixed, and docker suits those better than anything else I'm aware of—but if FreeBSD is in the mix, it's the odd-man-out. It's good not (only) because of its kinda-Jails-like isolation, but because of the excellent coverage of the image repository.
I don't use docker much so its not an issue for me. I could probably setup jails for the stuff I use docker for but I'm lazy so I have a debian virtual machine running on bhyve that runs any docker stuff I want. My preference is freebsd but im not precious ill use what ever tool fits the job best.
Also, Hacker News runs FreeBSD, for what it's worth :)
Also many firewalls use freebsd/openbsd, from pfSense to opnsense to genugate.
And Apple used it for their Apple AirPort:
I pretty much had the opposite experience.
https://qiita.com/tags/linux https://qiita.com/tags/freebsd https://qiita.com/tags/netbsd
So - your premise is wrong - but even if it wasn't, why be surprised? Some stuff is just fun to hack on, people have passions and interests too.
A predictable release schedule and praised upgrade paths mean companies can plan product updates long into the future.