Linux follows an inverse form of Conway's Law
medium.com
medium.com
The reason the driver subsystem is architected as pluggable modules ("drivers") is to support the extremely wide array of organizations that have to build into it.
The reason why Linux is broken down into subsystems is to support the "specialists" who work in only on system at a time.
The reason Linux is a monolithic kernel that has a large degree of complication internally (vs. a microkernel) is because Linus is strong enough to make it happen.
I mean, the logical error is right in the title. The author inverted cause and effect.
That is to say that subsytems were defined solely based on technical considerations, which is how it should be if the goal is sound engineering.
Not sure what to make of the ratio between "specialists" and "generalists". A comparison to ratios from other projects would provide some helpful context.
I think that's too idealistic. As another sibling poster pointed out, it's more like a democracy... with all the attendant upsides and downsides. (Also, I'm sure there's a lot of interpersonal politics involved, even if it's all over email.)
Don't get me wrong, the Linux kernel works surprisingly well and I rely on it for almost everything (including my livelihood), but if you really look at it, some of the subsystems are shockingly bad.
I think a good example is containers/namespaces which have a ridiculously bad security record. (See "user namespaces".) Again, I'm sure the people working on these things had the best of intentions, are very competent generally, and were hampered by the "never break userspace" rule. However, if containers/namespaces were truly designed, it could have been done a lot better. (See "Zones" on Solaris et al.)
The problem with namespaces and all the "container" tooling built on top of it is, as you noted, security. They were never designed to be secure from the start. It was not meant to be. And it was always puzzling to me why someone would use them to build containers on. Knowing it wasn't secure.
As you also note Solaris got it right with zones, FreeBSD got it right with jails, there was plenty of options to use and build on. But as is the typical Linux story they had to do things there own way, and poorly.
I still have no idea what Linux land is going to do for a filesystem. Since it's pretty clear ZFS is kept at arms length. Btrfs seems like the horse they are hitching to? Filesystems are hard. Which is why they can't seem to write a solid, reliable one and keep starting over. I think it's a big issue for them and is only going to get worse as data continues to explode exponentially.
I wish zfs had been BSD licensed. And I wish Linux had simply adopted it as well. But those ships have sailed. Where to now?
I don't know much about the BSDs, but my impression was that jails don't really offer nearly enough to rival Zones. Zones is really "containers done right"[1].
Definitely agree on ZFS. I'm currently using ZoL on my Ubuntu servers, but I'm not going to try using it on anything else. (FWIW, it's working marvellously so far.)
Personally, my current best hope for a good Linux FS at the moment is Bcachefs[2]. It's based on the proven Bcache. It's mostly done by a single very talented programmer/designer who knows when he's out of his depth[3].
[1] Not "virtualization" per se, but as close to it as makes no difference, if that makes sense?
[2] https://www.patreon.com/bcachefs
[3] https://bcache.evilpiepirate.org/Encryption/ (the initial paragraph; AFAIR there's a lot more out there about the process, but I didn't stash any links and that's all I could find in 30 seconds :).
I'm glad to hear your using it. I really can't imagine using anything else. Open source or commercial. Nothing is as compelling or reliable and easy to use. Me personally, i am forever grateful for Sun open sourcing it. I remember pre zfs days, with volume managers, and how hard it was to grow a ufs/ffs anything, just an absolute nightmare. And backups? Ugh. Tapes. Hoping every bit was copied, but never really knowing for sure. Just ugh. The pain.
I've heard bcachefs talked about to. And as you originally stated I think these folks writing btrfs and bcachefs are talented and are doing the best they can but again FS's are hard. Look at all the garbage we've been stuck with from all the major commercial vendors until ZFS. Everything Microsoft puked up, or Apple, even the major UNIX vendors sucked at making file systems and volume management. Veritas? I'd rather stab my eyes with soldering irons than use any of that stuff ever again. Seriously ZFS compared to all of them is night and day. I'll never be able to thank the developers enough for helping me out of the pit of filesystem and volume management hell. I'll be interested to see how bcachefs ends up. But mostly I think everyone will be trying to play catch up to ZFS and more than likely doing it poorly. But I applaud their efforts to try!
Thanks for the links btw, I always enjoy reading up on various tech.
[1] I tried to make this point, perhaps badly.
I suspect that even a NASA or an Apple or a Google or some other social structure would make a product with some painfully bad parts lurking somewhere in it.
I further suspect that the difference is, with open source code, the "general" public is actually able to become aware of it, rather than just the engineers working on it.
EDIT: Actually, before I forget, I heard a great aphorism the other day: When designing (distributed) computer systems, try designing it as if each "node" were an actual human. That'll usually ferret out most of the flaws in your reasoning. (No, it's obviously not foolproof, it's basically just a sanity check against assuming absolute reliability, etc. Kill cats. It's a rhetorical device meant to make you think. So think. Hey, do you own thinking. I can't think for all of you. Live your own lives!)
I know we engineers are getting high on it, but if you remove complex components, everyone else is much happier.
Really good complex systems thus, are seemingly simple systems, that cover with stupid components a suprisingly big number of uses-cases.
My argument is that there is no organizational structure that is more likely to product and maintain such a system.
People identify the bad parts of an open-source project and say "See? This is why we need Serious Engineers|Business Acumen|etc". Meanwhile, the bad parts of private project simply aren't made public.
That never was the goal and if it was it hasn't been achieved. Linux is like democracy, it isn't perfect but it is the best that we've got. Unlike democracy however it is fairly easy to get rid of Linux and replace it with something better.
Is it?
One interesting way in which you could do this is to build a wrapper around device drivers so that each device driver becomes a process on top of a microkernel.
The userland can mostly remain unchanged as long as you expose the expected interfaces.
Microkernels on the nixy side of the industry have so far been crushed beneath the tank tracks of path dependency.
Ah ok.
> Microkernels on the nixy side of the industry have so far been crushed beneath the tank tracks of path dependency.
If you shut those down society would grind to an instant halt. They're super successful just not very visible.
OS X is not really very adherent to the spirit of microkernels, earlier and more recent Windows releases have it well beaten in that respect.
As you say: It's all about steaming ahead on the mainstream[1]! Woo! /s
[1] I wish you both luck and success if you do something away from the mainstream. Because we need that, at least occasionally.
In principle yes, but Linux exposes a lot of interfaces, not all of which are that well documented in all their corners, and some of which are leaky in their abstractions in a way that makes it ugly to implement them on top of other systems. Stuff like: procfs, inotify, cgroups, Linux-specific syscalls like epoll, weird mechanisms like vDSOs, etc. The most heroic attempt I've seen to get all this working on top of another kernel is SmartOS's LX-branded zones, which are really quite impressive and implement most of what I just mentioned, although I believe even they are not 100% [1].
[1] These slides gave an overview of what worked and what didn't as of 2015. Not sure if there's a good updated ref, as I'd assume some things have progressed since then. http://us-east.manta.joyent.com/patrick.mooney/public/talks/...
Come on. You think that is the easy part? Just because everybody can hack up some "kernel" and boot their machine into it? I think you know the relevant Linus Torvalds quote in this context ...
The first can be done, I'm 100% sure of that, the second I'm not so sure.
I'm kind of surprised that on HN of all places something like a kernel would be considered somehow beyond re-creation or requiring special god-like powers.
Organisations are a superset of people, and demonstrate their own emergent properties, most notably organisational persistence beyond the founding remit.
People tend not to outlive their founding remit.
There's also the (general) case that kernel development doesn't create the self-propogating and self-funding structures you'll find within commercial (or even not-for-profit) organisations.
All of which suggests a possibly looser binding strength.
The possibility of an alternative development effort tackling a similar problem, under a different group and structure, and being considered not on profitability but on fitness to task likely also helps (modulo inefficiencies in assessing that fitness).
I believe the answer is, yes, it would. While Linus is a stubborn and opinionated leader ("Benevolent Dictator For Life") it is those qualities, coupled with his extremely high standards, that have preserved the coherence of Linux's system architecture all this time.
If non-hierarchical social structures are really more effective, such examples should be easy to find, no?
On the other hand, maybe their absence only indicates that online communities simply tend to mirror the social structures of offline communities, or that they're just mostly made up of people who prefer hierarchies.
https://en.wikipedia.org/wiki/The_Tyranny_of_Structurelessne...
tl;dr - hierarchies will form whether you want them to or not. If you refuse to endorse an official structure, you'll simply get an unofficial (and more often than not, unaccountable) one instead.
We do this at work. It mostly works, modulo "Distributed Systems Are Hard".
They are the kind of project that kills the recipient organisation.
My reply was about orgs, not your software. Since orgs are made of people, they are distributed systems.
The graph that would ultimately support the point of the article would have the difference between a simulation of a uniform distribution of contributions by the authors, and have a full 0-100% axis for scale, as opposed to the 35-65% presented in this article.
But in source control, author is typically defined as the first contributor to a file, which doesn't always reflect the person who contributed the most content to the file.
The very separation that the article draws "core" vs "drivers" is actually highly representative of how the Linux community is structured. Most of the core work (including the driver subsystem's backbone) is done by long-term contributors who actually work on the Linux kernel full time. Most drivers actually come from occasional contributors.
Driver contributions are "specialized" for the same reasons why they're specialized on pretty much any non-hobby operating system, namely:
1. The expertise required to write complex drivers mainly exists within the organization that sells the hardware. Needless to say, these people are largely paid -- by the hardware manufacturers! -- to pay drivers, not contribute to what the article calls core subsystems. There are exceptions ("trivial" devices, such as simple EEPROMs in drivers/misc, are written by people outside the organizations that sold them), but otherwise drivers are mostly one-organization shows. In fact, for some hardware devices, "generalists" don't even have access to the sort of documentation required to write the drivers in the first place. (sauce: wrote Linux drivers for devices that you and me can't get datasheets for. $manufacturer doesn't even bother to talk to you if you aren't Really Big (TM))
2. Furthermore, there really are subsystems in the kernel that are largely a one-company show and are very obvious examples of Conway's law. IIO drivers, for instance, while started by Jonathan Cameron who, IIRC, is really an independent developer, are largely Intel' and Analog Devices' -- to such a degree that, even though they follow the same coding conventions, if you've worked there enough, you can tell who wrote a given snippet. Same goes for most of the graphics drivers. Most of Infiniband used to be IBM, I think. If you dig down in the drivers subsystems, you'll see even funnier examples (my favourite example are ChipIdea USB controllers; a few years ago, support for USB slave mode on some Broadcom SoCs broke down because Freescale pretty much took over de facto ownership of the drivers, and some of their changesets worked fine on their ARM cores, but broke on Broadcom's funky MIPS-based cores)
Also, this is very weird to me:
> Adherence to Conway's Lay (sic!) is often mentioned as one of the benefits of microservices architecture.
Back in My Day (TM), adherence to Conway's Law was usually considered a negative trait, summarized by the mantra that, in the absence of proper technical leadership, an organization of N teams tasked with writing a compiler is going to produce an N-pass compiler.
Of course, this is a most negative example, but are we really, seriously considering that adherence to Conway's law is a positive thing today? That it's actually a good idea for the architecture of a software system to reflect the "architecture" of the team that created it, rather than, you know, the architecture that's actually best for what it's meant to do?