Debian developer prompted to revisit FreeBSD after 20 years
changelog.complete.org
changelog.complete.org
The one thing that surprised me was his take on FreeBSD's package management system, pkg. PKG is relatively new, having been released in late summer 2012. Prior to that, FreeBSD relied on the ports collection, which was (is) a vast tree of Makefile's allowing you to create custom builds of virtually any software imaginable.
While pkg is still raw on the edges, I VASTLY prefer it to Debian's hodgepodge of package management tools that all do 90% of the other, though none of them do it cleanly and none of them have straightforward interfaces. Am I using dpkg here? What about aptitude? Or should I just roll w/ apt-get? All of these tools combine the ease of use of git with the flexibility of a Maven build. If you know how to use to descend deep into the depths of git or Maven, this isn't an issue ( and for the author, it certainly is not ). Yet there are numerous simple tasks ( like say, searching for a package remotely when you aren't sure the exact name ) which still require hitting up google and settling in for a Click-Your-Way-To-Adventure session.
PKG, in contrast, is both simplistic and flexible ( it reminds me of a industrial grade version of Brew in some sense ). It's a tool that I find myself integrating into my workflow beyond simple installs / updates, particularly it's seamless integration with jails. Configuration of remote repositories is vastly simplified as well.
PKG is not perfect by an means - it definitely has worts that need to be taken care of. I'm also sure that PKG probably feels a tad inadequte to some hard core sys admins - pkg has the look and feel of a tool designed by a programmer looking to handle normal cases than a tool designed to provide a sys admin with an atom bomb if necessary. Yet the system is still relatively new and as the bugs continue to get ironed out it's a reminder to me of why I love FreeBSD in a lot of ways - the abstraction point is flawless and it gets out of my way.
I am considering writing a comparison command-to-command of apt-get vs. pkg, and differences in behavior.
* Are packages from the FreeBSD repos pinned the same way Ubuntu packages are, relase to release? Are there multiple repos (comparable to universe/multiverse, updates/backports)?
* apt-get update && apt-get upgrade
* apt-get --no-install-recommends install <pkg>
* apt-get purge; apt-get autoclean;
* apt-cache showpkg <pkg>
* apt-cache search <pkg>
* dpkg-reconfigure (for post-install scripts)
A pkgng repository can contain older versions of packages, but "/latest" is tree of symlinks to the most recent versions in the repo.
The FreeBSD Ports Tree isn't tagged, so there's nothing equivalent to updates or backports. Sometimes, that means packages won't build on older (or newer) versions of FreeBSD, or it means that the ports infrastructure as a whole won't work properly once you go more than a few releases back. The FTP archive keeps copies of binary packages included with past FreeBSD releases (all the way back to 1.0). Generally, that means everyone using the latest version of Firefox from ports on FreeBSD 9.x, 10.x, and 11.x are going to be running Firefox 35.0.1_1,1. The ports maintainer might include FreeBSD release-specific patches in the package build scripts to handle any differences among currently (or formerly) supported FreeBSD releases.
I don't remember what universe or multiverse contain. If those are the ones that have stuff separated out based on the software package's licensing, ports/pkgng includes logic to handle EULA acceptance for non-free stuff. You can also configure it to _not_ accept a license, in which case packages that use a rejected license will fail to build.
Regarding jails, please see "man pkg". You can give pkgng a jail name or ID, and it will automatically connect to that jail to perform whatever task you asked of it.
apt-get update == pkg update ("man pkg-update"), usually unnecessary
apt-get upgrade == pkg upgrade ("man pkg-upgrade"), automatically updates the repo metadata by default
apt-get purge == pkg delete <pkgname> ("man pkg-delete")
apt-get autoclean == pkg autoremove ("man pkg-autoremove")
apt-cache showpkg == pkg info <pkgname> ("man pkg-info")
apt-cache search == pkg search <pattern> ("man pkg-search")
dpkg-reconfigure does not have an equivalent on FreeBSD. FreeBSD in general takes a default-deny stance on things, so post-package setup usually has to be handled by the sysadmin. There are a few packages that do some interactive post-install setup, but I personally consider this a huge bug (e.g., mail/postfix). Port maintainers generally display post-install setup instructions instead. You can read these via "pkg info". I use the Salt configuration management system, personally.
As for documentation, the FreeBSD Handbook or manual pages are always the first places you should look. There's also some info about pkgng on the wiki, but it's pretty dated as at this point, all of that stuff has been merged into FreeBSD.
In ubuntu, main = FOSS supported by Canonical, restricted = non-FOSS supported by Canonical, universe = community FOSS, multiverse = community non-FOSS.
Those usually don't get updated or messed with except for security. There is also the backports repo for package updates for stable releases.
The FreeBSD equivalent of "main" would be FreeBSD itself, with source/binary updates via freebsd-update or source updates via Subversion. The FreeBSD Ports infrastructure handles package building for _all_ third-party software (Xorg, GNOME, KDE, Emacs, Perl, etc.), which would be the equivalent of "universe" and "multiverse". The Ports infrastructure has integrated license auditing (see https://raw.githubusercontent.com/freebsd/freebsd-ports/mast... and https://raw.githubusercontent.com/freebsd/freebsd-ports/mast...), which allows you to accept or reject licenses at package build time.
There is no equivalent to "restricted". The binary package repository maintained by the FreeBSD Foundation itself only contains free software, meaning that it includes only software whose licenses allow (a) redistribution of the original sources (like an FTP mirror), (b) selling copies of the original sources (like on CD-ROM), (c) free redistribution of the binary package (e.g., via FTP), (d) selling of the binary package (e.g., via CD-ROM), and (e) automatic license acceptance (which would exclude software like dccd that have EULAs which must be accepted prior to compilation/installation).
There is no equivalent to "backports" or "security". Supported FreeBSD releases get regular bugfixes and security updates as described above. As FreeBSD is a volunteer effort, there just aren't enough people involved to make maintaining alternate ports trees feasible. The ports tree does get frozen and tagged at each FreeBSD release to facilitate package building, CD-ROM manufacturing, and FTP distribution, but those tags aren't maintained except as historical markers. If you want security updates, you need to stay abreast of the latest commits the ports tree (whether via portsnap [source], subversion [source], or pkgng [binary]).
And it's worth pointing out that the per-branch repositories are all built from the same sources — ports is not branched. The only reason to have separate release-branched binary repositories is for ABI bumps between stable releases.
All pkg replaces is the old way of getting/managing binary packages (pkg_add, pkg_delete, pkg_info, etc).
My understanding was that pkgng is a architectural overhaul in addition to a simple client clean-up - configuration is different for example. What I was trying to get to was that the newer design is barely two years old in production. I should have written this more clearly and did not, thanks and sorry.
I could probably script it to just go ahead with the default dependencies, but that gives terrible results (I try to install this little console tool and now it's pulling gnome-something-something because the console tool can be integrated with ImageMagick and that's compiled with support for this-weird-format which is provided by this-weird-lib which is part of the Gnome project and now I'm slowly compiling half the packages in Gnome even though I don't use a single program from the Gnome project).
(I rarely install directly from ports, though. Instead, I use Poudriere to make my own package repository, and I have a version of make.conf set up for Poudriere that includes all of my package configuration tweaks.)
make config-recursive
a couple of times to do the configuration up front.
See also: https://www.freebsd.org/doc/handbook/ports-using.html
But it is not simplistic.
Just use "apt". It's new to Jessie, though.
There's a lot of people either threatening to switch or are switching to one of the BSD's as a result of being dissatisfied with the systemd debate outcome -- apparently not realizing FreeBSD (and others) are in the early stages of working on their own systemd/launchd-like init system. Everyone seems to realize the need for a more modern init system...
That's funny, because the loudest voices all seem to be insisting that an unholy shitpile of init scripts a la SysV was the One True Unix Way and had been forever (which shows how young they are, but I digress).
The number of people pushing for smf/launchd equivalents (let alone working on them) is tiny by comparaison.
The arguments I've seen are mostly that it's possible to write clean, composable, independent programs with solid, well-defined APIs between them, rather than the monstrosity that is the systemd project.
Want to replace journald with something else? The API for doing that is not stable, and I suspect the build system doesn't make it easy. Want to initialise kdbus under another init system? Even though it should have absolutely no dependency on systemd whatsoever, the component which does so is tied into systemd and can't be removed from it.
On the other hand, there's systems like nosh[0] which prove that you can build UNIX-y composable programs to solve many of the problems that systemd set out to solve. If you want to replace the logger, go ahead! If you want some component of it without using the whole init system, that's doable too. With only a little work, I'm sure it'd be possible to run much of nosh as non-PID-1.
(Note that I'm not suggesting anyone use nosh - the code is a tad weird and it's hard to find things, and the build system is extremely non-standard. However, it makes a useful proof of concept that the tightly-coupled systemd is not the only way to build something better than the SysV init system.)
[0] http://homepage.ntlworld.com/jonathan.deboynepollard/Softwar...
Event-based and dependency-based init systems are not controversial. Having an init system with inseparable and breaking changes to logging, IPC, etc. with its own DHCP client seem to be what people think is absurd.
systemd is a package of separable utilities, plus an init system. Much like the GNU core utilities.
The complaints about "doing too much" can never decide what too much means and usually start suggesting obvious, raw sore spots of SysV init are actually features.
> systemd is a package of separable utilities
It is absolutely anything but that.
If I want to use GNU coreutils with my own implementation of cat, I can write my own. I have to comply to the interface of GNU coreutils' cat, of course, but as long as I do that, I'm fine.
Good luck replacing journald with your own implementation. The API isn't stable or well-documented, so there is no interface that you'd have to comply with to speak of, and even implementing your own journal reader is officially discouraged.
I often find myself thinking that systemd (which I actually use on my desktop) would be far less hated if the development team would stop claiming things that aren't true. If I break a monolithic program into four parts that can't work without each other and can't be replaced, I'm only meeting a cargo cult definition of modularity.
What you imply is equivalent to wanting to run Gnome without libgnome.
If you want text logs, configure journald to pass things into rsyslog or whatever other logging subsystem you prefer (there are many).
>systemd is a package of separable utilities, plus an init system. Much like the GNU core utilities.
The response was:
>So, how does one completely excise journald from a box running systemd. I'm not talking about passing messages through, I'm talking about complete, utter, wiping out?
The point is that if the utilities are truly separable from the init system (as was claimed by the first point as an indication that the person they were responding to didn't "get" systemd) then that would be simple. Note the person responding is responding to the assertion that they fundamentally misunderstand systemd. Their point is to show that they do not.
Your first response adds nothing to the conversation:
>If you want text logs, configure journald to pass things into rsyslog or whatever other logging subsystem you prefer (there are many).
Good point, except that's not what the person you're responding to asked for.
My observation is that systemd isn't just "an init system with separable utilities" that was originally claimed. You seem to agree with me.
Your last point, however, would have been helpful above:
> Anyone could implement their own journaling subsystem against the api and it would work... it doesn't have to be journald.
That would have answered the actual question asked.
Instead of implementing your own reader or writer for journal files we ask you to
use the Journal's native C API to access these files.Apparently proof it's too monolithic is literally 1 dependency between two parts of it
I might give it another look once the internal API between all of the core components is well documented, isn't changing every 17 seconds, and any work to rip out or replace components doesn't require you fight tooth and nail with the build system.
The theoretical possibility of systemd not being monolithic by way of tightly-knit incestuous dependencies doesn't mean much when the reality is that the deck is stacked against you if you try to replace any of those dependencies.
I'm not even against something being monolithic or 'different' - I run a ton of illumos based servers, I love SMF, and am looking forward to the FreeBSD implementation of a more modern and featureful init system.
(I'll disclaim I don't like Lennart or Kay - I've found their behavior to be ridiculous in quite a few ways, and that certainly colors my judgement... but my dislike for them stems from behaviors in relation to technical problems, so I wouldn't say said dislike is without merit)
The fact that it took four releases to add the setuid/setgid statements kind of tells you all you need to know about how many sysadmins the developers talked to.
I don't think that's at all true. It's not been any of the arguments I've seen (e.g. it wasn't the case made by boycottsystemd.org ). If anything that sounds like a strawman from the systemd side.
2) "which shows how young they are" -- what in particular is your issue with older, more experienced people, and how is that a problem enough for you to call it out?
Which, it turns out, are wholly underwhelming for many things people now want their init system doing. Hence launchd, SMF, and Jordan Hubbard's interest in an equivalent. But I guess the BSD developers working on the former and latter don't know what they're doing either, right?
> what in particular is your issue with older, more experienced people
I'm sorry you're struggling to read, but it's quite the opposite: anyone around long enough knows SysVisms were derided widely when they were introduced.
Shitty init scripts and the path to systemd have been self-inflicted wounds caused by a tiny group of inexperienced desktop developers. Such is life when you live in a local minima caused by your own incompetence, I suppose; even terrible, tasteless directions look like improvements.
Oh, you mean like Red Hat (the largest enterprise and server open source company on the planet)?
What about SUSE, Debian, Ubuntu... the list goes on...
It's hard to take you seriously when you're not even wrong.
You're creating a false dichotomy. Just because SysV-style initscripts are a mess doesn't mean systemd is the only viable option.
IMHO, I wouldn't count on any major init changes in FreeBSD any time soon. It's been protracted enough already, and Jordan Hubbard giving a pep talk doesn't mean all that much. I could very well be mistaken.
The differences in scope between launchd and systemd are pretty big, too.
There were several half-serious individual efforts, but I don't think the community / foundation were pursuing it seriously until quite recently. Kip Macy has been working hard to get it in as an init replacement recently.
> It seems like progress with openlaunchd is going well, but there are certain things related to Mach IPC that need to be ported to the upstream FreeBSD kernel to get it working.
The mach stuff has been ported (recently), again by Kip Macy.
> I do recall some old version of launchd entitled launchd_xml was ported to pfSense, but it's one that puts the XML parser into PID1, so it's not desirable.
Right. I think the current plan is that the parsing is done in another process and then IPC'd to launchd via Mach. Or the plist stuff could be replaced with a 'libucl' parser that is safer in init? I'm fuzzy on these particular detail, sorry.
> IMHO, I wouldn't count on any major init changes in FreeBSD any time soon. It's been protracted enough already, and Jordan Hubbard giving a pep talk doesn't mean all that much. I could very well be mistaken.
It's going to happen soon. :-)
> There were several half-serious individual efforts, but I don't think the community / foundation were pursuing it seriously until quite recently. Kip Macy has been working hard to get it in as an init replacement recently.
This is because iXsystems (Jordan Hubbard) hired Kip to implement it for FreeNAS (and, presumably PC-BSD), not because the FreeBSD community thinks it needs a new init system.
Jordan also has plans to bring other large pieces from Apple to FreeBSD.
I believe EMC Isilon has also sponsored the work.
> not because the FreeBSD community thinks it needs a new init system.
Right. Kip is working on it because someone hired him too. But practically, this is how large-scale efforts get done. Some FreeBSD-using corporation hires / builds out a larger project in a way that is acceptable to the community and it is incorporated into base.
I do think the community has come to more or less that consensus at this point, though. The camp that is fine with /etc/rc hasn't started raising loud complaints about the idea incorporating a backwards-compatible launchd to replace rc.
It has kqueue, which is a generic event mechanism that can take the place of inotify, signalfd, eventfd, timerfd and others on Linux systems. It has a bit of a learning curve, but it's actually neat.
Concerning platform support, that has always been NetBSD's shining ground.
And on Linux inotify is bound by /proc/sys/fs/inotify/max_user_watches. Six of one, half dozen the other. FreeBSD file descriptor limit is arbitrary and can be raised to INT_MAX (2 billion). That should be plenty.
> i must say that the kqueue implementation was by far the worst to work at.
I agree that the kqueue API is a pain to work with. I haven't tried to do a recursive file monitoring (dropbox-alike) application with it.
It seems easy to implement inotify as a thin wrapper around kqueue. Maybe something to do for the FreeBSD linuxulator layer...
Yes, on Linux you're limited to max_user_watches, and you can edit that value, just like you say with kqueue... but you need root access, so for end user applications this is not really a solution. With inotify i use one watch per-folder, with kqueue you need to do it by file ( of course you can watch the folder changes, but you need to keep a copy of the file states in that folder, and then re-stat that files to find out what file was changed, not even close to an ideal solution ). So for example in my use case with inotify i can watch 65536 folders, and with kqueue less than 10000 files ( that could be a single folder! ).
Edit: It seems that FreeBSD default a much bigger number for FDs ( https://news.ycombinator.com/item?id=9063893 ), so that should suffice a lot of use cases, that's good news for me!
Ah, I looked at inotify API briefly and did not notice that directory watches return file changes. That does make it easier to use than kevent vnode watches. Mea culpa.
Edit: It seems to be set dynamically depending on available memory at boot. See http://fxr.watson.org/fxr/source/kern/subr_param.c#L266
May be it was PC-BSD, i don't have it installed right now, so i can't say it with certainty. But that number looks just enough.
You are wrong.
ulimit -Hn; ulimit -Sn 467631 467631
Now try that on RedHat/CentOS as a default non-root user.
Here's the link: https://www.freebsd.org/cgi/man.cgi?query=kqueue&sektion=2
You want to watch a directory? Open it and register it with the VNODE filter. You can watch a file for delete, rename, extend, write etc. OpenBSD can even watch TRUNCATE events.
You can use a kqueue filter for userspace events between threads, timers, signals, process states, asynchronous io, writability, and readability of fds.
It's quite generic and fairly evenly supported across BSD derivatives including Mac OS X.
Linux hasn't lost its way, it has evolved over the past 20+ years as UNIX has evolved over the past 40+ years. So have the BSDs--all of them.
but there is no way with FreeBSD, like there is with
Debian, to say “never even show me packages that aren’t
DFSG-free.”
See /usr/ports/Mk/bsd.licenses.mk (https://raw.githubusercontent.com/freebsd/freebsd-ports/mast...) for the gory details.
pkg metadata does include license information -- see https://github.com/freebsd/pkg/issues/20, but I am not sure of plans to provide user-facing control over acceptable licenses at the moment. It's a good idea though and I'll see if we can make it happen.
Looks like the author is still learning FreeBSD and hasn't read all relevant docs yet. 'man 8 gvinum' clearly says that raid5 _is_ supported:
"To create a raid5 array on disks /dev/ada1 /dev/ada2 and /dev/ada3, with stripesize 493k you can use the raid5 command: gvinum raid5 -s 493k /dev/ada1 /dev/ada2 /dev/ada3"
I'd take this article with a big grain of salt.
The interim solution is RAID6, which will work for some years to come.
But my point was that:
1. You still have some lost data.
2. An RAID5 array which breaks in this way must be manually rebuilt, possibly from a rescue boot media. Your server will no longer even boot up by itself.
That's definitely a feature, not a bug! I hate how Linux distributions require you to install a thousand "-dev" or "-devel" packages. All headers should always be present on a Unix system!
Also, you don't need HAL/PolicyKit/dbus on FreeBSD. xorg-server recently switched the default configuration option from HAL to devd. And you can compile things like Firefox WITHOUT="DBUS" :-)
If you can repackage without breaking the upstream-supported use cases then sure, go for it, if you want to.
If it really is a problem for you then work around it. But for most of us installing dev files makes sense if we ever need to compile something against those .h files.
Except Archlinux (and Gentoo, probably).
>Its storage subsystem also has some surprising misses. Its rough version of LVM, LUKS, and md-raid is called GEOM. GEOM, however, supports only RAID0, RAID1, and RAID3. It does not support RAID5 or RAID6 in software RAID configurations!
Well, actually, it does support raid5, but also... With ZFS being a truly first class citizen, why not just use raid-z or zfs encryption instead of raid5 or luks?
>ZFS also does not support some common operations, like adding a single disk to an existing RAID5 group (which is possible with md-raid and many other implementations.) This is a ZFS limitation on all platforms.
No block pointer rewrite for ZFS, so no expanding of existing vdevs by adding disks, but you can still add more vdevs to increase the total pool size. You generally want specific numbers of disks based on raidz level for performance reasons anyway.
(In general) RAID isn't really a "consumer" thing, outside of maybe a basic home NAS type environment. If you're doing this for a server environment where RAID sees the majority of it's use, I don't think it's too onerous to have to add more vdevs instead of a single disk.
> Apparently this is because FreeBSD has nothing like Linux’s inotify.
Nothing like? kqueue/kevent. It's not 'exactly' like inotify, but it's definitely something like it. (It's also something like epoll, and others... kqueue is fairly big and replaces multiple linux equivalents)
I find it odd that the network stack isn't mentioned either. It's still significantly more performant than the linux network stack, and FreeBSD is pulling away even farther in this arena.
Things like https://wiki.freebsd.org/NetworkRSS , netmap being a first class citizen, etc (Netflix blogs on why they chose FreeBSD for their CDN devices are fascinating) are all pretty damn sweet.
Kind of nitpicky, I guess, especially since the author isn't claiming to be intimately familiar with the state of every part of modern FreeBSD
It also seems to me that most of the issues mentioned would be non-issues for OpenBSD (though it does not support ZFS, which the op wants).
What does he mean by this?
However, the points which he thinks are where Debian is stronger illustrate precisely where Linux has lost its way. Laptop support is lacking -- assuming you care about laptops, and care about suspend working out of the box. Guess what, suspend didn't work out of the box under Debian, either.
Storage, he's forgetting that zfs is both a filesystem, and a volume manager.
Support of alien file systems, really? Irrelevant for servers, for everything else mount it via NFS.
Virtualization -- between jails and Virtual Box (on desktops) what is the use case for a different product here?
Nothing like Dropbox -- that's not a bug, it's a feature.
Desktop environments need configuration -- why I am migrating from Linux, you think? If I already have to use alternative window managers and desktops on Linux, where is the big difference?
Package managers, now that is a good point. XML config, that too. Nigh everything else is missing the point.
But, to address the questions: suspend does work out of the box for many laptops on Debian, including suspend on lid close, and has for years.
ZFS as a volume manager - I specifically addressed this. ZFS is not a suitable volume manager for everyone because it is fairly rigid once a zpool is set up. You cannot expand a raidz1 nearly as easily as a RAID-5 in Linux. (You have a stripe the data across another storage group.) You can't shrink a zpool at all. These are limitations that matter in some, but not all, setups.
Support of alien filesystems may be irrelevant for many servers, but it is not for all. But anyhow, people use an OS for more than just servers.
Virtualization -- believe it or not, there are people that need ways to run Windows. Sometimes a whole lot of Windows.
Dropbox -- lacking something that can to instant auto-syncs is a feature? That's pretty presumptious.
There is a big difference on desktop environments and configuration. Install Debian and with a simple selection of "desktop environment", it boots up into a working DE. FreeBSD -- not so much. Install a bunch of packages, exit some polkit and hal files, and then you'll get there. Well, if you thought to run pkg inside typescript so you see what pkla files to edit.
What a horribly arrogant and conceited thing to say.
* You run Tarsnap whenever you want to push changes; Dropbox automatically pushes changes.
* Tarsnap is for 'real' computers only; Dropbox runs on my phone (which is very useful for photos, among other things).
* Tarsnap is billed by usage (at a very small amount); Dropbox has a free tier.
* Tarsnap doesn't have a natural sharing usecase; Dropbox does.
* Fundamentally, Tarsnap is backup, and Dropbox is sync. They are two different things. Not to mention that Tarsnap is also supported just fine on Linux.
I dunno about that, but there's always OwnCloud which runs anywhere you can get PHP running (yeah, 2 minute hate on PHP, who cares it works.) and has clients for iOS, Android & everything else you can think of