A Generation Lost in the Bazaar (2012)
queue.acm.org
queue.acm.org
As believing something is possible is a pre-requisite to actually approaching it, I expect things to continue getting worse.
One of the biggest problems I’ve had with hiring engineers out of big companies is that they’ve developed a sort of learned helplessness. The longer they spend in big, slow companies the more they’ve developed an idea that everything they do must adhere to mountains of process and approvals and documents.
It can be a challenge to un-teach these ideas from people after hiring them into a company where they don’t need to put together presentations and seek multiples levels of approval before doing anything.
One surprise that I didn’t account for is that many people come to like these process-heavy systems with approvals everywhere and no progress until the rules have been followed. It becomes a safety net to pull as many different people and process into a decision before action is taken. If something goes wrong, no one can blame you because you followed the process and got all of the approvals, right? I think this leads a lot of people down the road of preferring high-process, low-action environments because following the process is a type of low-effort, low-risk work whereas producing something new is high-effort, high-risk.
So no, I do not think this is about "low effort", though that can be a side-effect. No one sits in an office wanting to find ways to make their coworkers lives difficult. More often than not the policy is a response to pressure to do something to prevent whatever failure prompted the policy. The people who like the process heavy approach, in my experience, anyway, are often remembering a time when failure to follow a procedure resulted in negative consequences for someone.
That’s not exactly what’s happening. It’s not about making coworkers’ lives more difficult. It’s about preferring pseudo-work that looks like progress (meetings, process, consensus-building, planning sessions, endless documents) instead of work that delivers results.
Once a project ships, everyone can see whether it was a success or failure. However, if you can keep it in the planning limbo for as long as possible you can avoid that moment of truth. Another round of meetings and refactoring and consensus building is safe because it’s easy to say you’ve followed the process and therefore produce good process-work. Shipping to customers is risky because then the product can actually fail.
In companies where someone’s reputation and promotion prospects depend on avoiding project failures, it becomes attractive (and safer) to deal more in the pseudo-work than the actual work.
Isn't this almost all companies? Remembering one's success and other's failure better seems to be human.
"Jim really fucked the project up" vs. "Our project was a great success" (junior dev Jim getting no mention).
More charitably, they feel this work will make the product better. That may or may not be true, but if you've been involved in failures and part of the after action review was "this person should have been consulted because they are the expert in whatever failed", that becomes part of the process. Pretty soon, this refactoring and consultation becomes expected, and so on until we're at the point you are speaking of.
And I say this as someone who is part of the problem. I have unfinished projects right now that are stuck in meetings and consensus building. I want, at minimum, to be able to say, "you were given the opportunity to participate in the process, but you left it to me." but I've also made mistakes and poor decisions for lack of those steps more than once.
I have no doubt that there are people avoiding failure by not shipping, but I do not think it is as simple as that, there are a variety of cultural norms and experience colliding here.
AFAIK, most people mainly focus on what they do and deliver, as it's what matters the most, but many fail to consider that how they do it might matter as much. Yet, knowing how every aspect of the environment impacts the efficiency of the work would be priceless. Being able to continuously improve and adapt the work environment so that the work keeps being done in the best possible way? One can dream ;)
The technical debt concept somewhat applies to the organizational field: what may have worked well for the small startup is almost always re-worked once the company grows, and even when the company no longer grows in size, but still in age, frequent maintenance is required for the business to stay relevant in this changing world.
Obviously, how to design how good work is done is no silver bullet, but as someone said: “I would rather have questions that can't be answered than answers that can't be questioned.” Like the not-to-be-further-questioned answer I once got: "Look, that's just how things are done around here." ;)
So, from the employee perspective complex process means a lot of work to do (job security), higher pay (even if the domain problem is not too difficult process complication makes it difficult reducing the available supply of qualified labor) and less responsibility if something goes wrong.
From the business point of view compexity means, being current, keeping up with the latest tech, signaling you are not a dinosaur. Keeping up with security because the shoddy earlier layers had holes because software engineers do not output a finished work product like other engineers. Even if some dinosaurs may do things cheaper and better.
From the customer perspective the incentives are pretty often the same. Many customers want a touch screen instead of a knob or button because the latest tech all has touch screens. They want to use Slack or VS Code because that's what the cool kids do these days.
From the outer space perspective it looks like humans are building a new tower of babble with their tech stacks. Ever higher layers of complexity not actually going any place or doing much new.
Eventually it might all crumble into a pile of confusion.
For me personally I have a bit of a dislike for GNU software in general because of the same type of things. It's all autotools and legacy code being dragged around. Don't get me wrong, I fully appreciate the GNU project and how much they have done; I just don't like their code very much.
On a larger scale the Bazaar suffers from not having managers paying people to do boring things. This leads to boring things generally not being done as much, people on their own wants to work on interesting things.
Granted, developers often do this, too. If you can find a way to align incentives with users, I'm all for it.
I've started reading the Autotools manual and turned away, to much bloat and hard to read code. Meson popped up and I immediately went that direction.
That does definitely not work, see enterprise software
Sure it means I would run into complications on a 30 year old AIX system with some old defunct C compiler, but that's a tradeoff I'm willing to make to simplify the build process.
If you aren't including headers from the "linux" directory a BSD system is almost identical to a Linux system as far as C is concerned. Common libraries like OpenSSL, zlib, libpng, and so forth are identical as well. pkg-config exists on modern BSD systems and takes a lot of the guesswork out of dependencies.
The pre dot com days were not a golden age of software quality.
He also has this idea that Unix is a "beautiful cathedral". No, the descriptions of the original environment where Unix was designed resembles to a internal AT&T version of a bazaar. Multics was the well architected, beautiful cathedral.
The Linux kernel looks to be a successful Bazaar project with a single person (Linus) having nominal responsibly.
I recently return to UNIX after a 20 year break. Linux in 2020 is much better than Solaris in 1998. However, Windows made giant leaps as well. Both still have massive amounts of deadwood but Windows does a better job of hiding it.
Depending on your purposes, Linux in 1998, was better than Solaris in 1998, even on Sun hardware.
There are some great stories about the engineering they've done to continue to support legacy applications (because those applications matter to customers).
"One of Brooks's many excellent points is that quality happens only if somebody has the responsibility for it"
Linux is a prime example of this with Linus. I fear the day he's not around as a guiding force.
I think that all the article is self disqualifying. Maybe this was acceptable language in 2012, but it was still having a bad effect on people, arrogance cannot substitute argumentation.
A Bazaar in conjunction with regulations is more sensible. As an architect at a medium sized organization my job is not too build pretty cathedrals, but to allow the teams in my department to row in the same direction. I set non functional requirements, I set direction, but I cannot know the systems that each team is developing as well as they do. Bazaars require rules, but they evolve dynamically with the needs of the organization.
I love the sagrada familia un Barcelona, but it's still under construction after 150 years. Building cathedrals maybe not the way.
Well, they are not jus nailing two pieces of wood.
Also, if I were to build a cathedral today I would use 3d printing and CNC milling machines as much as possible.
A real cathedral is a monument for the ages, bigger than any individual who worked on it. Something for entire generations to pour their effort into so that at the end of construction you can say: "my grandfather started building this, my parents worked on it, I worked on it and during the time of my great-grandchildren this building will still be as magnificent as it is now".
Trying to build it as quick and cheap as possible is simply not the point for a cathedral.
I wonder how many other shops there are where implementations aren't designed to last long, especially in web development.
Nor do they flourish inside supermarkets, these buildings are designed for something else. Walking in St. Peter's in Roma gives a sensation of awe even to devout atheists, something I can't imagine the local Walmart doing.
For awe, I would much more recommend the Pantheon, especially knowing that you're stepping into a building for which 500 years is barely more than a quarter of its history.
Even further off-topic, but if you are referring to the big church in the Vatican adjacent to St. Peter’s Square, it is Saint Peter’s Basilica; it is the second-ranking of the four major basilicas but it is not (though many tourist-focussed lists of cathedrals include it) a cathedral.
The cathedral that is the seat of the Bishop of Rome (and also the first-ranking of the four major basilicas) is the Archbasilica of St. John Lateran.
One of the emperors that rebuilt it still kept the "Agrippa did this" inscription, you don't mess with Augustus.
That's the whole point of cathedral projects - make it beautiful and build it right because you're building for the future. Not to scratch your own personal boredom itch, or to nail a bit of wood to another bit of wood because why not.
UNIX may not be the best example for your argument: it's often criticized because of how long its legacy has endured, with some people arguing it's time to move on to more modern visions of what an operating system should be. So UNIX, for good or bad, is indeed the software equivalent of an ancient cathedral which endures from ages past.
Which is to say, engineering culture and goals can absolutely degrade over time in terms of productivity, even with great increases in construction technology.
Interestingly, this may also be a sign of declining engineering. The Hagia Sophia, the largest church in the world for 1000 years, was apparently built in 5 years (532- 537).
In general, it seems Rome was able to build fantastically persistent buildings in fractions of the time required later. Even today I wonder if we actually have the engineering and logistics practices required to build buildings that would survive thousands of years in a under a decade - of which the Romans had several.
Or it could simply mean that not enough resources were committed to the project.
I find it refreshingly honest, vivid, cathartic and spot on. Complaints from the peanut gallery just make me love it even more.
In 9 or 90 years, maybe they'll think all of our 2021 eggshell tiptoeing to be completely useless drivel devoid of insight and dump it down the memory hole. Sensibilities change!
I can't help but think all articles like this are fighting a very old and outdated strawman. I'd venture that literally no one is arguing that "Free Software is magic" anymore.
Look, sometimes Free Software is the thing that works well, sometimes non-free but open, sometimes proprietary is what you have to use. There are a lot of things we can work on and do better, and concrete solutions and experiments are important to consider: but this sort of big picture whining isn't helping anyone.
I'm a huge proponent of improving software Quality. I have found that complaining about it doesn't actually do much good. My complaints tend to get downvoted into the Mantle, and I just get pissed off.
So, my approach, is to do the very best I can, in my own work, and limit the amount of other people's work on which I rely. When I do use it, I am quite careful, and vet the work. Even so, I have had to back out several highly-rated packages that I had thought were good, but my apps started crashing, as soon as I incorporated them.
Just mentioning that, has often proved incredibly unpopular. It seems to be considered blasphemy.
It does reduce the scope of what I can accomplish, but I am quite happy with what I do.
> I have found that complaining about it doesn't actually do much good. My complaints tend to get downvoted into the Mantle, and I just get pissed off.
IMO creating real change in an organization is almost never done through complaining, it’s done through pushing practical solutions in a positive/constructive way. It might not be the content of your argument, but the way you present it. This comment, for example, instantly biased me against your argument, because the tone felt complain-y and self-righteous. I think regardless of what your argument is, if you present it with this tone, you won’t be persuasive.
Software Quality is not a popular topic for discussion. HN isn't too bad, but some other communities react to Quality discussions, as if I was insulting God.
I've learned to just keep my opinions to myself. Even then, when I talk about my own experiences, and my own process, it is not received well.
We used to stand in a field where everywhere we looked we could see and hear another of us, now there's 10,000 fools between each of us, and it's rare for us to hear or see another over the crowd.
If your argument comes off as “I don’t use libraries because everyone else’s software is crap, so I write everything myself,” people aren’t going to respond well to that. But if you can make a convincing argument for why it makes sense to avoid dependencies in specific situations, I’d imagine you’ll get a positive reaction.
But I have had issues, when incorporating the work of others. I do use a couple of dependencies, in my work; but just a couple (besides the default ones, like standard libs, and platform SDKs). That's a big reason that I don't use cross-platform frameworks, like QT or React Native. They are fine, but I feel that they reduce the quality of the user experience. I've tried using them for decades (cross-platform frameworks are nothing new), and have never been happy with the result.
I am always nervous about running the critical path through dependencies. I worked on a project, a few years ago, that was completely based around a particular tool; assuming the tool would improve, and that our implementation of it would improve, as we learned more.
Neither happened. In fact, the more we learned about the tool, the more we realized how unsuitable it was. That ended up being a real disaster. Basically flushed a couple of years (and several million dollars) down the drain.
So I tend to stick with native, and I also write my own dependencies. I believe in modular architecture. It does have a few issues, but these can be greatly mitigated, if I have control of the whole stack.
I come from an embedded background. I have worked with computers, right down to the FET junctions, and I like to know what's going on, everywhere. I'm uncomfortable relying on stuff I don't understand. I've learned to relax, over the years. Compiler toolchains are really quite good, and I don't understand them (in detail). I don't understand AI/ML, and I'm not quite to the point, where I incorporate them without question, but I think we'll be at that point, fairly soon.
I'm kind of amazed that a lot of folks google a dependency, then add it as a critical path component, without a second thought. I guess it has positive results (meaning that money is made) often enough, but I consider that living a bit too dangerously, for my comfort.
On the flip side … some of the worst software mistakes at places I’ve worked at have been of the “let’s build some common thing ourselves” variety. For example, for some crazy reason, someone implemented their own HTTP-esque protocol on top of ZeroMQ, complete with servers and clients in multiple languages, that was used in a massive number of services. A total disaster, insane amounts of dev work invested in it, tonnes of bugs and performance issues, eventually we ripped it out and replaced it with HTTP, but it took literally years of work to do so. Less extreme were in-house service discovery tooling, and an in-house version of something like Redux. Tonnes of investment to build them, poorly documented with poor test coverage and lots of bugs. Then the main author/advocate leaves the company, it becomes a huge maintenance burden, and eventually it’s ripped out and replaced with a popular open source version at great cost/effort. In all cases, the in-house version was massively lower quality than the top open source alternatives, in basically every conceivable way.
I agree that introducing a dependency is a significant decision, you should research the dependency well and not add it lightly. But building everything in-house can be a major mistake with massive unnecessary costs, you need pretty specific use cases and the right company culture for it to work out.
A level-headed comparison of actual historic Unix sources, which are publicly available for study, reveals this to be false. Unix was always "just hack it", and later Unix is, overall, an improvement over earlier Unix.
It seems I have not come across test cases in old Unix code bases.
Can someone point to some circa 1980 Unix code base where there test cases? For anything: system calls, shell, yacc, cc, ..
I think the idea may have been that if Unix rebuilds from scratch and you get to a system prompt where you can type "ls", the toolchain has been validated.
Something about treating something which ain't yours (such as a software system with millions of lines of code) as if it were.
Something about maintaining a complex software system which you are responsible for, but somehow are never allowed to make simpler for your own sake. You are only paid for work which increase the "value" (money, profit, whatever) of the real owners of the business. It's not really worth their money to pay you to make your own job easier.
> Quality happens only when someone is responsible for it.
The easy way for this to happen is to be able to own, really own what you do. This just doesn't happen in the 'software industry'.
Who owns the product? The company.
Who maintains the product? Whoever the company pays to maintain it. Usually, companies will build out trees of responsibility so some sub-group is responsible for smaller pieces of large products (with ultimate responsibility "rolling up" back to company ownership).
How can I contribute? Your boss will tell you what needs maintenance or improvement. Do it or you're fired. If you have something else you want to work on, a good company will respond to your feedback and shuffle responsibilities around (you get better work when employees are satisfied), but ultimately, someone needs to maintain the build tools and if that's what they hired you to do, that's what you're doing until you decide to stop taking their money for it.
There are lots of disadvantages to the corporate model, but (like the old chestnut from Thomas Paine's writings), it has the advantage of simple ownership.
The rumors of open source's demise have been greatly exaggerated.
I use open source both privately (Linux, some open source games, FreeBSD a long time ago), and professionally (mostly Java and database stuff), and while sometimes I can't help rolling my eyes, on the whole open source works incredibly and surprisingly well; especially compared to the alternative of closed-only corporate software.
1) insist on PIC (what Linux does)?
2) replace all pointers with dummies and fix them up at load time (what Windows does)?
3) always load a given library to the same place in virtual memory (what Linux did in the a.out days)?
Before you have a single compiler flag for dynamic library generation, you have to answer questions like this. And no single answer existed in the Unix environment of 1990, which wasn't a bazaar -- it was more like a struggle among competing cathedrals! Hence libtool, whose messiness reflects the messiness of the Unix scene at the time. GNU actually made monumental efforts to clean up the different Unixen and make them more consistent.
Fortunately, today, all widely used Unixen these days seem to have settled on ELF format and option one -- all, that is, except macOS!
I think it's unfortunate that people only skim over it and then use it to complain/defend Linux/windows/m4/autoconf/etc. More important is the main point that "quality only happens when someone is responsible for it". This explains a mistake I see time and time again in all manners of software projects: anything by committee, anything overly democratic, can only aim to be mediocre at best. You need an owner/architect/champion/tyrant if you hope to achieve anything above average (not guaranteed though).
> there is no standardized way to build a shared library in Unix. Instead of standardizing how to do that across all Unixen—something that would take just a single flag to the ld(1) command
I've never looked into it, but I'd love to know what ld flag PHK was thinking about. Could the entirety of libtool still be replaced by that flag?
I think we need to push for more quality in Open Source software, and in commercial software. That quality can come from a team, though it seems more likely to be driven by a charismatic BDFL. The quality though comes from tests, attention to detail, QA, etc.
Poul Henning-Kamp has been around long enough to see the end of his system and the rise of what Raymond talked about; high priests like Henning-Kamp just don’t matter as much as they used to, compared to the insane productivity of the entire world pumping out stuff whenever and however they want, gluing together whatever they need.
There is a lot of shit being made, sold and given away in the Bazaar, but there are a lot of gems. A tech startup cost millions of dollars to start in say, 2008, the height of the dotcom one boom. For reference, discount hosting providers charged on the order of $30,000 a month for a single Sun server to serve your website.
Today, a solid startup can glue together hundreds or thousands of open source libraries and launch product for free. Even if you ignore free time from the AWSes of the world, many a startup can structure their infra around serverless functions and be paying in the 10s of dollars a month. That’s the power of the bazaar, or whatever system we have now, which is more complex than Raymond described.
I think the vast majority of users of this site do not want to return to a world where releasing products takes someone of Henning-Kamp’s intelligence and skill - there just aren’t that many of him in the world, and we would have far fewer awesome things in that world.