A Generation Lost in the Bazaar (2012)

The horror shows I have witnessed and had to work with inside some of the companies that have employeed me over the last 15 years have been much worse than anything I have come accross in reasonably large and active FOSS projects.

The Peter Principle lives on strong in both camps.

edits: spelling

Sure you could have a single person responsible for quality within a project, and that may even be a good idea (BDFL is a pattern for a reason). However you can't do that across organizational boundries. Ports will have lots of duplication because the entire point is to bring the wider world together including projects competing against each other.

Like the solution just doesn't seem applicable here. It feels like the equivalent of someone suggesting to fix the problem of firefox & chrome doing the same thing by appointing a single person responsible for web quality. The solution doesn't seem wrong so much as non-sensical.

I don't particularly see the connection to cathedral & bazaar either. Even before that book competing software projects had duplication.

Some 'law' about always something, somewhere depending on whatever got built previously. Documented or not.

Of course there's the rewrite. Read: starting over with a better design than existing one. Yet another skill that few mortals possess when it comes to intricate machinery like an OS.

And then a whole bunch of functionality must be re-implemented. Which takes countless human-hours (always in short supply). If not years.

That leaves the option to replace pieces one by one with better ones, leaving overall structure (mostly) in place / working. Oh wait, that's what is usually done. Rip something out, plug the hole, move on.

This immediately reminded me of Helm charts (Kubernetes) and their implementation with Go templates, that work in the syntatic level instead of semantic, which makes it unnecessarily hard to operate it.

Few forces favor convergence and standardization in open source. After a few decades of this, we have far too much duplicative stuff.

It's really hard to clean that up. At one time I tried to get the 6 or 8 Python packages that parse ISO8601 date strings standardized. All of them had bugs. It took 6 years of bikeshedding discussions.[1] Issue filed in 2012, patch applied in 2018. For something that's a few screens of code.

[1] https://bugs.python.org/issue15873

But, the reason for the described situation is no mystery, and may not have that much to do with the bazaar. Programmers like to reinvent stuff, as well as use what they’re familiar with. Those two facts alone explain a lot of the criticisms.

I suppose, in a properly managed cathedral, these tendencies might be somewhat mitigated. But how many cathedrals are properly managed, or have the luxury to “do things right”?

The whole “worse is better” debate is relevant here. Worse may not be better because it’s actually better, but because it’s more easily achievable, particularly by people without infinite time, expertise, and money.

But it's probably worth pointing out that while cathedrals are built, with one architect's vision, not all people are capable of envisioning a cathedral. Many see only ugly 4-walled-flat-roof sprawl. They might build it really well, but its still sprawl.

Equally, things improve with practice and time. My early work may be "single vision", but I'm more skilled now, my vision is clearer and my execution of that vision is improved.

Wonderful software, like wonderful buildings, are wonderful partly because it is not common.

Good vision is partly driven by talent, partly by experience - good software is the result of both great architecture and a capable team willing to work in service of that vision.

Of course all good builders want to be architects, and few want to be remembered as the successor to someone else's vision.

The problem of duplicated dependencies in linux comes from doing too little splitting of packages, not too much. You get a lot less of this on a platform like nodejs where people properly split up functions into much smaller libraries, because the dependency management is much more effective. (Of course people on HN moan more when you do that, for no discernible reason, as if having 1000 dependencies is somehow morally inferior to having 10).

Ultimately a certain amount of duplication is necessary and healthy. There is a nugget of real criticism to be found here - the "Bazaar" nature of open-source OSes makes it very difficult to deprecate and remove anything, where a more centrally-managed system is able to make decisions like "we'll stop using or supporting Perl from 2020" and enforce them. But "herp derp 122 packages herp derp 22 tests" is not what we should be focusing on.

His beef is that people are lazy. If there is a perl tool to munge something you use it, even if the rest of the process uses Python. This is the reductionism of "if you start in awk stay in awk" which is good, honest and .. hard.

Ports is better than APT. apt, has the disease of 'I made a bundle for you but the name is not indicative of whats in it' -where Ports has metaports which are explicit what they pull in to make the "thing" they meta over.

APT has the "-dev" norm now. thats ok, but sort of stupid too. How can it possibly make sense to have to pull in the -dev thing to use its shared libs standalone in another package? (I have seen this)

Ports is the bazaar but with sub-zones. Like the Grand Bazaar in Istanbul, things tend to clump together. Sometimes you wonder why its in sysadmin/ and not text/ but in the end its birds-of-a-feather.

Homebrew ditched M4 for Ruby. Gak.

CPAN is the enemy.

I remember great pieces of software from before the Internet era. The grammatical tense that applies to them is "past". They are dead and gone and only exist as "hey, remember Code Warrior?" "remember the LISP Machine?" "remember Lotus Notes?" Millions and millions of dollars, hours, lines of code all just gone. They live on as inspirations but not as artifacts.

The "cathedral" model doesn't produce cathedrals, it produces photos of cathedrals that were abandoned and eventually collapsed and were forgotten.

The bazaar has produced code that outlives its authors, its sponsors, its original communities. You can Tanenbaum Linux all day long but it will outlive the cockroaches. That's the real monument.

In the article, PHK seem to assume that it’s a good idea that each user compile all their applications locally. I know this is the norm in some places (e.g. Gentoo?), but is it really a reasonable expection of users in general? I consider myself a power user, but if I want to install a tool, in most cases I just fetch it using Homebrew and I will get a binary, so no need to compile it at all. Compilation and packaging might be complex for some of those tools, but it’s “out-sorced”. IMHO a system such as macOS + Homebrew works well from the end-user perspective.

Do I say that just because I’m lost in the bazaar? Have I even seen one of these cathedrals that PHK refers to? Not sure. What would count as a cathedral? HP-UX? Symbolics’ Genera? Would OpenBSD count?

Strongly disagree. We need to get back to the utopian ideals of the early web where normal people were encouraged to cobble together weird and wonderful websites.

Seriously though, the problem is that the Cathedral is a Utopian fantasy that has never once materialized in our timeline and the people who attempt to instantiate it invariably create something much, much worse than the Bazaar.

It may look like it's made of stone, but those arches are made of painted poo.

Sure, the Bazaar is ugly and even offensive to the refined eye of the great priesthood, but these things are almost always far more useful because they come from necessity, closer to the person with real needs.

So true, I've yet to see anything of quality produced by groupthink.

Autoconf doesn't exist because "the bazaar" is full of lazy incompetents and only cathedrals have halfway decent design. It comes from the bad old days when "open" meant "spec documents are available under RAND terms" and had little to do with what we now call open source. Unix had splintered into lots of different idiosyncratic systems, each with its own configuration, and the GNU stuff had to run on all of them. Autoconf provided a portable way to paper over all those differences.

The same with libtool. The Unix splintering had already happened when shared libs became relevant for Unix. There wasn't even agreement among the various vendors on how to implement shared libs! For example, how do you resolve the fact that a dynamic/shared library could be loaded anywhere into a process's address space, making absolute pointers inside the library invalid? There were two main schools of thought:

* when linking the library, replace all exported pointers with placeholder values. Then when the dynamic linker loads the library, it swizzles the placeholder values back into pointers. (Windows)

* use position-independent code, taking advantage of relative jumps, calls, and data accesses to create an object file that can be loaded anywhere. (ELF)

I've seen some half-measures as well:

* give each library a fixed address where it's loaded every time (Linux a.out)

* punt on symbol resolution; hand the programmer a base address and have them call into the library at fixed offsets from the base (AmigaOS through 3.x)

Each of the implementation strategies for shared libs required a different constellation of compiler and linker flags. Again, libtool papered over these and made it tractable to write shared libs for a variety of operating systems. It is true that these days, compiling with 'gcc -fPIC' and linking with 'gcc -shared' will probably do what you want. But that assumes a modern enough OS and toolchain, something that GNU couldn't and still cannot in some cases necessarily assume.

The purpose of both these programs is to portably solve problems that were created by multiple, competing cathedrals. I don't like Autoconf or Libtool. They're unholy messes. But they're largely that way out of necessity considering the problems they solve, problems which would otherwise be nigh-intractable -- not because their developers were lazy or incompetent.

And no, Meson does not solve the same problem, it punts on working on any OS that's insufficiently modern.

The neat thing about the open source bazaar is that it can produce reference implementations for things everybody agrees should be in the finished product, reducing the friction of competing implementations. An example is the 86open consortium, an effort by various proprietary x86 Unix vendors (mainly BSDi, Sun, and SCO) to agree on a single binary standard for x86 Unixes. Well, the vendors got together fully expecting to get into the weeds with meetings and committees and stuff... but what they found was all their products had Linux/ELF kernel personalities already built in. So they declared Linux/ELF to be the binary standard and disbanded the 86open initiative!