Kernel – Fix cluster_write() inefficiency | Hacker News Reader

Kernel – Fix cluster_write() inefficiency

gitweb.dragonflybsd.org

84 pointsprotomyth53 comments

Looks like HAMMER is DragonFlyBSD's filesystem, for those (like me) who may be unfamiliar with the distro: http://www.dragonflybsd.org/hammer/

Previous discussion about the filesystem vs. ZFS: https://news.ycombinator.com/item?id=8829194

Not really fair to call it a distro, you get the wrong impression, its was forked of FreeBSD 2002 or 2003

Distro is synonymous with OS at this point. No one is using it as a derogatory term for OSes that aren't different enough to give them the denotation of "OS".

That said, you're going to have to come up with an argument stronger than "this was forked ages ago" to suggest Dragonfly and FreeBSD are different enough to warrant thought. Ubuntu forked before 2004 and you don't see anyone being called out for comparing it to Debian.

> Distro is synonymous with OS at this point.

Possibly to the layman but the term "distro" (short for "distribution") deliberately exists to differentiate between different OS's and different distributions of the same OS but with a different software stack and default configurations.

While you do sometimes get differences between the Linuxes in terms of init daemons and slightly patched kernels etc, they are all generally GNU/Linux - ie generally share the same common Linux OS fundamentals (eg GNU coreutils). There will be exceptions to this rule (isn't there always?!) but we're talking about the common desktop / server platforms people automatically talk about when discussing "Linux".

However many of the BSDs are developed in insolation. While they may have a shared heritage, the kernels have matured into something quite different from one another. Often the core utilities and init daemons et al can vary noticeably as well.

So in essence, FreeBSD and Dragonfly are different but similar OSs, whereas Linux distributions are the same OS but differently configured. This is why we make the distinction between "distro" and "OS".

I've been experimenting with Debian/kFreeBSD lately, and one funny aspect it's had is making me realize that the kernel is not really all that important to the userland experience, at least for what I do. It still feels like Debian, even though they swapped out the Linux kernel for the FreeBSD kernel. That's one reason I think of Debian as an OS, rather than a Linux distro. Debian itself then has several distros, such as Debian/Linux and Debian/kFreeBSD.

I get your point, but aside the package manager and associated tools, most of what you've described are GNU packages / cross platform user land. ie stuff that even in the most limited sense was never Debian specific to begin with; but often was designed to run on most Unix-like platforms anyway. Because of this I've read some people describe kFreeBSD as GNU/kFreeBSD (ie GNU user land with the FreeBSD kernel) and I think that makes some sense if you're following the GNU/Linux naming convention (which I think makes sense in the context of this discussion because it also takes the Hurd kernel into account - GNU/Hurd). But this is one of those edge cases I was thinking of when I talked about exceptions to the rule in my grandparent post.

As for whether it's classified as a separate OS from Debian (Linux) or FreeBSD - there's definitely some room for interpretation so I'll hold off from passing my own personal judgement :)

Talking about rule exceptions, another good example would be Android. Largely the same kernel as GNU/Linux and some of the user land too but equally it's a very different platform to "desktop / server Linux"

One big difference is that Debian has invested a huge amount of work over the years cleaning up the software they package to make it easier to customize, maintain configuration across updates, etc. That's much broader than the percentage of GNU utilities in Debian's userland.

Debian aren't the only ones that do that however patching 3rd party software doesn't make it "Debian's user land".

Debian also runs a lot of software written by Redhat (eg systemd) but that doesn't make it a distribution of Redhat. Ubuntu ships a lot of in house software as well (eg Unity) but that doesn't mean ArchLinux with the Unity DE turns Arch into a distribution of Ubuntu.

What you're doing is akin to classifying groups of web browsers by the websites the user visits rather than by the rendering engines they're built on.

I find it more integrated in Debian than in most other distributions (even including the BSDs, once you get outside their base systems and into the wild west of ports). It's not just that they ship other software in the package manager, but that it's heavily customized to fit into the "Debian way" of doing things. This is partly emphasized by the lack of a distinction between "base" and "ports"; every package, from libc to your mailserver, is a Debian package, and is supposed to follow policy and work how users expect a Debian package to work.

It does vary package to package, but on average there's a fairly substantial amount of work that goes into "Debianizing" a package so it's integrated into the OS coherently, vs. the more lightweight build-scripts that you find with systems like Slackware or FreeBSD ports, that typically ship something closer to upstream. I would compare it more to maybe halfway towards how FreeBSD adapts third-party software into its base system. FreeBSD base is full of third-party patched stuff, too, like ZFS, LLVM, OpenSSH, and sendmail, which is periodically synced with upstream. But they put enough effort into customizing it and making it work together coherently, that I think it's fair to call the base FreeBSD install an "operating system".

I do think Debian as an OS may get less true with systemd, though. SystemD is really going all-in on the idea that there is a specifically Linux way of integrating an operating system, which is closely tied to the kernel, and Debian increasingly finds it difficult to avoid getting pulled in 100% to that path, since the amount of divergence you need to avoid doing things the "systemd way" is growing rapidly. In which case the end game is that there's a Linux/SystemD core OS, of which Debian is just one distribution. While until now Debian has aimed at being a "universal operating system" not tied to any specific kernel.

FreeBSD is a distinct operating system.

I don't agree with your feelings towards Debian simply because I've used plenty of platforms that have the same cohesive feel. But at the end of the day all you're talking about now is an emotional impression based on anecdote, which may feel relevant to yourself but ultimately has little impact to the discussion of distributions Vs distinct operating systems.

It might be more useful to see distro as orthogonal to OS. Is Debian/Darwin a thing yet? :)

If you use Linux with systemd or without is a significant difference, which is probably as big as switching between BSDs.

The name of my operating system is "Ubuntu", not "Linux".

> probably as big as switching between BSDs

No, it's not. You can go from one GNU/Linux distro to another GNU/Linux distro and you're still gonna find a GNU userland and a Linux kernel. Base userland and kernel differ significantly between BSDs.

plus - native binaries can be copied between one and the other and run natively without any emulation layer provided the c library, kernel version, and other libraries are reasonably similar.

> If you use Linux with systemd or without is a significant difference, which is probably as big as switching between BSDs.

I did address the init daemon point. With Linux distros you're still using a common kernel upstream (ie Linux) which all Linux distro's share. That's not the case with the BSDs. You're also sharing the same GNU coreutils which 99% of Linux's share, BSD's often have their own. The kernel is really the key point here though.

> The name of my operating system is "Ubuntu", not "Linux".

Technically it's neither. "Ubuntu" is the distribution name. "Linux" is the kernel name. I don't much agree with Richard Stallman that "Linux the OS" should be named "GNU/Linux" but from a technical standpoint it is a more accurate OS name than either of the two suggestions you raised.

To be fair, D in BSD stands for distribution.

Hehehe it does. But that's more a historical artefact as BSD originally started life as a distribution of UNIX.

We are talking decades ago though. Long before FreeBSDs grandparents, let alone Dragonfly.

edit: Wikipedia has an interesting piece on the history of BSD: https://en.wikipedia.org/wiki/Berkeley_Software_Distribution...

DFBSD has substantial kernel differences from FreeBSD (IIRC the original reason for the fork was that the author disagreed about how to implement multiprocessor scheduling and went off to do it his way), such that the diff between the FBSD tree and the DFBSD tree would be enormous at this point. Drivers sometimes make their way between the two, I believe, but it isn't just a copy-paste operation.

Ubuntu and Debian's kernels, on the other hand, are kept as itemized patchsets against the same kernel, and it would be hard (but probably not impossible) to find drivers that build+work on a Debian kernel and break on an Ubuntu kernel of similar age.

An OS is a kernel and userland developed together in its entirety. All the code shipped is maintained by a single responsible entity: an open source team or a company. External code is imported into the source tree and treated as if it is owned by the team. Any patches required for it to integrate cleanly are the responsibility of the OS maintainers.

A distro is a Linux kernel and a collection of open source software, typically GNU, which fulfills the needs of the userland environment. The distro maintainers are responsible for packaging and distribution. They are not responsible for maintaining the kernel or any of the userland software unless it was internally developed such as a custom installer or various glue code, maintenance scripts, etc. They do not own the libc, the kernel, the shell, any drivers, responsibility for writing security patches, or bug fixes for any code they did not originally write, etc. The distro volunteers or employees of the company in charge of producing the distro may be involved in kernel or other open source softeare development, but it is not required.

This may be an unpopular statement, but a Linux distro is not an OS.

It is a collection of open source software that tries to be compatible with all of its components. Subject matter experts of all components are usually not involved in its creation. Instead they exist spread out across the entire Linux distro ecosystem.

Does Arch Linux have an expert in the virtual memory subsystem of the kernels they ship?

Does Gentoo have an expert in the kernel random code?

Does Debian have an expert in the glibc codebase?

Does Ubuntu have an expert in the bootloader?

Does Mint Linux have an expert in the BASH code they ship?

I can say with confidence that Suse and Redhat employ experts of many areas of the kernel, glibc, systemd, filesystems, and other components, but not everything.

Does FreeBSD/Dragonfly/OpenBSD have an expert or designated owner of the filesystems they ship? Bootloaders? Libc? Kernel timekeeping code? Random? Crypto? VFS? Network stack? NIC drivers? CPU schedulers? Shells and userland utilities?

Yes they do. They're OSes, entirely maintained in-project. Just like Windows, OSX, Solaris, etc.

OpenBSD is most aggressive here, but unmaintained code usually means nobody is using it and it will be removed. Everything has an owner and is constantly being looked after. In the BSD world it would not be common for a major component to exist long term without an SME joining the project and maintaining it or replacing it.

BSDs are OSes, and are mostly independent. You cannot easily port kernel code between them because they have diverged significantly.

This is nonsense.

Except it's not. If I hit a bug in the NFS code, who at Debian will help me? Nobody. The bug report is opened and the buck is passed upstream. Now they have to hope the upstream maintainers of the NFS code care to fix it. They might not be interested especially if the distro is shipping an older release they no longer wish to maintain.

If I hit a bug in the NFS code on FreeBSD, I open a bugzilla issue first, but get the attention of Rick Macklem. If I hit a bug in the posix shell, I contact Jamie Gritton. If I hit a bug in the VM subsystem, I want to talk to Konstantin Belousov. A weird timekeeping issue? Most likely our resident Time Lord, Poul-Henning Kamp. A C conformance issue? Presumably David Chisnall. Even our compilers have patches required for our codebases, so we have compiler maintainers. If I hit a bug I have a team of people to reach out to.

All of these people have a vested interest in bugfixes and maintaining the code within our supported releases. You do not have this luxury in the Linux world. At best you can hope you have a support contract with RedHat or someone who can pay a competent employee to figure out how to fix it if upstream refuses to get involved.

I do like those qualities in an operating system. But they're not a requirement for the label.

I maintain that System and Distribution are not compatible by definition.

Distribution: A spatial or temporal array of objects or events.

System: A group of interacting, interrelated, or interdependent elements forming a complex whole.

I admit I am not an expert in language, but I have yet to see an explanation of how these two terms mean the same thing.

> An OS is a kernel and userland developed together in its entirety.

[citation needed]?

How is a Linux distro not an OS? Isn't it the Operating System on whatever machine it's running?

If I run Ubuntu or Arch, what OS am I using?

> If I run Ubuntu or Arch, what OS am I using?

Linux.

Well, that's a kernel. I don't see why something needs to be either a distribution or an OS.

Ubuntu is an operating system. It is also a distribution of Linux.

If I'm using Debian with the Hurd kernel, is the actual OS Hurd? This makes no sense.

You are right about the kernel point however there's more to it than that.

Debian is a collection of software and configuration that's distributed for convenience. That's where the term "distro" originates. The OS would be better described as GNU/Linux (the GNU user land and Linux kernel) with "Debian" being the name for that particular set of user land defaults.

Likewise Hurd is often referred to as GNU/Hurd since it too uses the GNU user land. Debian, Arch, etc will then take the Hurd / Linux and GNU sources and add their own touches to it (package management, etc) but fundamentally Ubuntu and Arch are still the same platform at their core in a way that Ubuntu and FreeBSD is not.

I didn't agree with much of what the GP raised (particularly around issues of support), but one point he was correct about was how Debian, Arch, etc aren't the maintainers of the Linux kernel nor GNU coreutils. Sure they will contribute patches but essentially what they do is redistribute GNU/Linux with their preferred packages and configuration. Hence the term "Linux distribution".

So the TL;DR version is:

If you're running Debian you do have an OS installed. The OS might be referred to as GNU/Linux to express the GNU user land and Linux kernel. Debian is the name given to the distributed collection of programs and default configurations that's based on GNU/Linux.

You're gonna have to let the Debian people know about that. They've been calling themselves an Operating System for as long as I can remember. :-)

Now you're running Android - what OS are you using?

That was against HAMMER 1, which is completely different to HAMMER 2.

Reading the diff, I'm neither surprised that this bug existed nor will I be surprised when similar bugs arise.

There must be a more readable to write even low-level code like this.

Yes, there is. That code needs iterators.

but most of that code isn't iterating. the only parts in that function i see that are iterating ([0] and the similar loop below it) seems straightforward enough to me.

i don't see how they would help here unless i don't know what you mean by iterators (assuming c++-style iterators or similar).

[0]: http://gitweb.dragonflybsd.org/dragonfly.git/blob/cf297f2ccc...

honestly, i think it's hard to say this generally without more context into the code. imo nothing there looks too egregious (or at least not abnormally so) for very low level code.

Couldn't they dump out the list of all individual write commands sent to the disk and have seen that the writes are in a suboptimal pattern? This would have been one of the first things I would have done when testing a file system of any kind: Make sure the thing writes what I want to be written and in the way I want it to be written.

People write bugs occasionally.

You're talking about the person who spent months tracking down what everyone else thought was a code or compiler bug but actually proved was a subtle CPU bug requiring vendor errata releases, and further, who wrote this log-structured, snapshottable, syncable low-memory using, real-world filesystem from scratch for the highly parallel operating system he has been rearchitecting for over 10 years now. There are consistent status update posts outlining the on disk filesystem structure, as well as its relation to the virtual memory subsystem and the in-kernel parallelism feeding IO writes, as well as constant code commits implementing these things, so I'd reserve the need to back-seat drive here unless I could back it up with a comparable body of work.

http://www.theregister.co.uk/2012/03/07/amd_opteron_bug_drag...

You don't even need to do anything that hard. All you need to is run basic benchmarks and compare the results to what other filesystems can do, and when yours is about half as fast that should make you go "huh"...

Hammer is twice as fast, not half. So huh

So it'll now be 4x?

We need benchmarks, and this is the old version only.

Many problems have obvious solutions in hindsight.

Isn't NCQ meant to do this? I mean reorder the write commands into an optimal order.

Always wondered where the bandwidth was going when I was using HAMMER. Had to use UFS when testing last time.

How often does a sequential write happen in an SSD ?

As often as it happens with HDD, I think.

Anyway, with my SSD:

I unzipped a 300 MB Raspbian lite zip into a 1.3 GB image yesterday and extracted parts from a 8 GB video with mmpeg last weekend. I expect all of them to be sequential given enough free space on the SDD (same for HDDs.)

I also work with docker and vagrant and building those images at half speed can't be good.

Almost always if one wants to optimize...

wow, this is an embarrassing bug. To say it doubles bandwidth makes it sound like an optimization, when in fact it is just (finally) doing what you think it should be doing. Yikes- how long was this doing this and no one noticed?

Data loss and corruption is always a lot more embarrassing. Did they have tests that ensure correctness (in data) of the slow version? Then they were already delivering on quality of the file system.

So if I'm reading this patch correctly it's now writing things in blocks instead of as single bytes? Is that it?

And, well, yes. Ofcourse that does wonder's for throughput. And I would certainly hope that someone who writes a file-system knows that.

Edit: Re-reading the patch it seems like what's being done here is taking a (failed) at attempt at writing in blocks and actually making it write in proper block-sizes. Yay C, I guess.

Wow, somehow you managed to read the patch without reading the commit message that explains what actually changed.

No, you didn't read it correctly.

Well yes actually. The fault is completely on me.

But in my defence, the code and diff is all colourful and highlighted... While the patch description is "hiding" very subtly in plain sight at the top.

When someone is new to any kind of system, their eyes will follow what attracts their attention. I didn't even notice there was a commit message there until you told me. Some minor UI tweaks could probably improve that page greatly :)

It's amazing how good we are at filtering information at all levels. I agree with you that the design of the page plays a large part. GitHub probably had more time & reason to optimize this.

Yes, they should increase the font size of the log section. It's as simple as

    .log { font-size: large; }

using their CSS class and their conventions for font sizes.

Reply on news.ycombinator.com