Do not ship work in progress: An open letter | Hacker News Reader

Do not ship work in progress: An open letter

dont-ship.it

161 pointssellweek75 comments

For some additional context: https://blog.brixit.nl/why-i-left-pine64/ Comments: https://news.ycombinator.com/item?id=32494659

> Supporting Manjaro has historically done very little to facilitate the development of the software stack which is necessary for these devices [Pinephone/Pinephone Pro] to work. In some cases the Manjaro involvement actually causes extra workload for the developers by shipping known broken versions of software and pointing to the developers for support. Which is why https://dont-ship.it/ was started.

So distros like Manjaro should stop calling themselves "rolling release" distros and start calling themselves "nightly" distros, so everyone is aware of the potential instability?

Is there any rolling release distro that already follows the suggestion of only distributing tagged releases?

Distributions like Manjaro should change their behavior and quit shipping unfinished patches.

What if the whole point is about shipping unfinished patches?

But also this should mean that the buck stops with the distribution, not with the developers.

What is the use case for shipping unfinished patches?

Depends what you mean by unfinished. Does it work and need a style polish? Is the developer trying to debug some tests failing for reasons unrelated to the patch? Is the functionality still missing/broken? There can be lots of reasons why you'd want an unpolished patch rather than wait for a release. Especially with projects that release every few months rather than after each pr.

Here's one example where I've done it: https://github.com/NixOS/nixpkgs/blob/350fd0044447ae8712392c... since new rust has already been merged and rbspy still has no release with the required patch, there are two options: a broken package or an unreleased patch.

It's a cost/benefit calculation for the maintainers.

In the case of manjaro which triggered this post, it's not that manjaro is fixing things. It's manjaro trying their absolute hardest to ship new features faster than the rest, even when those features are completely broken.

Like the fun example of manjaro shipping a merge request of the SMS/chat application on the PinePhone that causes a database upgrade that can't be rolled back. This code was not intended to be merged, even less shipped.

Yeah, it's a balancing act, and it's best when the users and maintainers are on the same page: I don't like maintainers futzing about with packages to fit to arbitrary standards (debian can be quite bad here, and the general policy of 'as close to as vanilla as possible' is something I like about arch), but on the other hand it's much appreciated when packages come with fixes for bad upstream decisions (of which there are many examples), or even important features which upstream has sat on for years (see pulseaudio support for high-quality bluetooth codecs for an example which eventually led to the patch author rage-quitting the project because they strung him along for years). As a user I'm generally happiest when I get the software with the features I want without the bugs which cause me trouble, and it's situation dependent whether the packager or upstream are working against me on that. Usually the biggest source of friction is who actually winds up supporting the result: for all that distros may encourage filing bugs on their own tracker usually users wind up going to upstream for support, even if the issue is caused by the distro's patches. Some badly behaving maintainers (and it sounds like manjaro have been far too aggressive in trying to pull in new features) have I think caused a general pushback from upstream developers as they wind up with a bunch of support requests from someone else's screw-up (see for example home-assistant's attitude to anyone else distributing their software).

If I understand your example, you are using your good judgement as a developer/maintainer to release a workaround that ideally would have been released as a bug fix by the maintainers of the root cause. This is not, however, the issue here, which is the judgement-free release of every work-in-progress as soon as it is made available to anyone, with the bag being foisted on the developers. If that solved your problem, and did so without introducing other problems, that would have been just a matter of luck.

I'm guessing that what you did fits within the article's guidelines so long as you could do it as a tagged release.

I understand the desire to get an early warning of breaking changes coming from elsewhere, but surely scraping together all work-in-progress is likely to raise a lot of false concerns?

> What is the use case for shipping unfinished patches?

I think the issue isn't an individual patch that hasn't been completed, such patches are unlikely to be checked-in anywhere.

It is patches that are part of unfinished work, or that have not been integration tested with the rest of the product, or are not stable/useful/safe without other work that has not yet been merged. Basically patches intended fr other project devs, not the general userbase yet.

But to answer the question directly, in case it is literally unfinished patches:

> What is the use case for shipping unfinished patches?

“Being the latest & greatest” willy waving in end-user aimed distributions.

I can see a use case for some dev communities, though mainly for proactive compatibility testing purposes, not for dev or non-test deployment environments.

> What if the whole point is about shipping unfinished patches?

Then make absolutely sure your users all realise this, and that they should come to you for support not the devs (unless they themselves are devs, have read the relevant documentation, all of it, and able to offer useful help with the matter).

> But also this should mean that the buck stops with the distribution, not with the developers.

The clarifying points further down the article, that is the thrust. The basic rule for releasing WiP code being “don’t” and the advanced rule being “don’t, unless you really know what you are doing and can support it yourself”.

If the point of your product is shooting yourself in the face, you should find a different point. Not all ideas are worth pursuing.

> start calling themselves "nightly" distros

It's not a "nightly" distro, it's a "let's take patches that are unfinished, release them into the wild onto unsuspecting users, and let the upstream developers deal with it" distro.

Which, to be fair, is pretty much what all upstream developers have been asking for for years and years because getting bug reports for 50 different builds of 15 different versions of your software all with different disto provided patches is exhausting.

The software that actually has good release hygiene pales into comparison to the software that is actually more stable by just pulling from main.

It is generally not a problem, distributions are doing absolutely great things and I like working with distributions a lot more than dealing with something like flatpak.

It's just that some distributions make a huge mess of it, don't want to work with developers and actively push off all the work to the developers for patches that have not even shipped, not even been merged or most painfully: not even been reviewed yet.

> Which, to be fair, is pretty much what all upstream developers have been asking for for years and years because getting bug reports for 50 different builds of 15 different versions of your software all with different disto provided patches is exhausting.

Then don't do 15 different versions of your software. I ask again: Why do i need GTK1, 2, 3 and 4 on my system ? Why KDE 5 is not compatible with KDE4 ?

Versions != major releases

No. They asked distros to ship their latest release as default version instead of recreating Frankenstein's monster holding onto end of life releases for ages backporting random patches without understanding the code they're messing with and blaming the upstream projects when their patches inevitably introduce more regressions and less features than any well maintained upstream project.

Of course this is a very one sided view. I've been on the receiving end of reckless upstream projects far too often not to understand why distributions which are expected to maintain compatibility with releases for years dislike a fast moving upstream for anything too important.

Manjaro Linux is accused of something different far less justifiable: forking upstream projects in all but name by shipping heavily patched packages without supporting them and even worse putting the support burden and blame on the upstream projects when things break (and oh boy do they break).

I believe the term is 'cowboy' distros.

openSUSE Tumbleweed

I always thought that the whole point of Manjaro was to be the "more stable" Arch linux, since it holds the rolling updates and releases them at once every month. Is that not right anymore?

I've been using Manjaro for quite awhile and have yet to have an issue with a broken update. YMMV I guess.

Yeah, but I can understand that the experience of the users (who generally don't see any issues) is different from the experience of the application maintainers (where the few users that do see issues report those issues).

I think this is asking that we avoid additional episodes like the Redhat gcc "2.96" issue where redhat took cvs head, and slapped a version number on it that would have been the next release and shipped it with their "stable" distribution. GCC devs got tons of complaints about their shipping a broken compiler that couldn't compile the kernel and tracked it down to redhat shipping pre-release WIP software as if it were the new release version. GCC completely skipped the 2.96 version number, so "2.96" ended up being a redhat exclusive-- the fake 2.96 would have been a recurring source of pain due to confused redhat users making bug reports to the gcc project.

https://gcc.gnu.org/legacy-ml/gcc-announce/2000/msg00003.htm...

Is there any dev methodology that doesn't get bogged down by process?

> Is there any rolling release distro that already follows the suggestion of only distributing tagged releases?

Void Linux

I worked with a "brilliant" product manager whose idea was to onboard several of our enterprise customers right after our first major deliverable, ie. midway through feature development. I vehemently pushed back, saying that it would be disruptive to our customers since the feature wasn't fully finished and it would slow us down, because we would need to change the order in which we would do development since customers expect a certain level of quality. I also said that any timelines after customers were onboarded were at risk, because if things were buggy, which they probably were since the feature wasn't finished yet, it would mean we would have to jump on them since they were our biggest customers.

These all fell on deaf ears because they thought it would be important to get early feedback from our customers. I told them we could demo it, but we shouldn't onboard them. Again, they refused to listen.

Things ended up being exactly as you expected, and I quit the job so that I didn't have to deal with this PM any more.

What does this anecdote have anything to do with the article?

> I worked with a "brilliant" product manager whose idea was to onboard several of our enterprise customers right after our first major deliverable

FWIW, I recognize it's fun to target PMs for ignoring technical constraints & carrying water for marketing... but for many PMs (in USA at least), they roll up to Marketing dept, rather than Eng.

So while it can seem PMs are (willfully?) technically Invincibly ignorant by default, their bosses are worse.

The best way to 'manage' your PM is help them build the biz case for your position. Eg "reduce risk of $XX loss" from bugs, opty costs, network effects of customer losing faith in your product. Plus I've found the "walk before you can run" argument works: they want to expand customer excitement by showing bright/shiny/new things. Promise them an even faster cadence of new things, after they give you time to get the fundamentals deployed.

Funny how I was instantly triggered as a SaaS maker by the title, ready to unroll in the comments and then quickly realized upon reading that I am not the audience here and the title does make a lot of sense for the intended audience :)

Unexpected self-calibration completed. Nice.

Same, I thought it was going to be an argument about MVPs

Many startups wouldn't ever be able to get started under a restriction like that.

“If you’re not ashamed of your first version, you shipped too late.” Reid Hoffman

This is bullshit.

There is a big difference between:

1) Shipping low features but high quality SaaS aap: OK

2) Shipping low quality services with lots of features that are buggy: NOT OK

From a customer standpoint, I reject half-baked SaaS services that reek with lack of quality control. If you are ashamed of your first version because of bugs, stop and fix those. If you're ashamed of how minimal your first app is? That's fine. Make sure those features are high quality and work as intended.

Spoken like a developer, not a businessperson.

The real world is a lot more complicated. As Rumsfeld put it, "You go to war with the army you have, not the army you might want or wish to have at a later time."

A startup might have a product with a promising and unique core feature, but still have many other features that are buggy. It's common for startups to be overambitious, and that often manifests as buggy features.

Startups like this will often have a high churn rate with early customers and trials. But that changes over time as they get experience with which features and which issues matter.

If you want to kill a new business quickly, "stop and fix" the bugs in your first version. The problem is, your first version may very well not be the one with the best product/market fit, and you just wasted your precious investment money, time, and resources fixing bugs that ultimately won't matter.

To be honest this is a very click-baity title that doesn't have the context to make it a truth. A lot of us probably came here expecting something else, like you.

A better title would be "Do not ship someone else's work in progress"

This applies even to games. Although it is normal now for games have early access releases, it is becoming common for rushed 1.0 releases and then patching coming later... coupled with a ton of negative reviews, backlash and lost sales.

I wonder why publishers don't realize people expect the product to be done when you remove "beta" from its name.

They see companies which have great success despite consistently shipping broken products for decades (Bethesda), and lots of indies shipping unfinished products through proper expectation management (early access), and think they can get away with it. Often there's also great pressure to hit particular time windows (e.g. release October-December to get Chrismas sales, release live at a trade show, avoid releasing just before or after a more popular title in the same genre, etc)

So if there’s a patch that fixes a devastating bug then distros should ship with the bug, got it.

Or reach out to the developer who does not respond to email (who is likely also not a signer of this open letter and who may or may not agree with it).

Multiply this (futile) reach-out step times however many developers are involved in touching any code of any project being shipped during any if the multiple days, weeks, or months between releases.

Which is probably hundreds of unanswered emails. Mmmkay.

The page literally addresses this:

> We thank all the distribution package maintainers for backporting patches that improve security, fix bugs, etc. who coordinate with upstream. Often times this means creating or pulling patches to fix issues with inactive/abandoned/unresponsive upstream projects. These distribution package maintainers are doing a tremendous job and their work is not the subject of this letter.

> This letter wants to address the cases where actively-developed features, huge changes, etc. of active upstream projects are being included without the knowledge of the project maintainers or end users.

I wouldn’t say it addresses it so much as it acknowledges it as an issue without offering a workable solution.

What's a more workable solution for urgent patches than not including them in the request to "not ship work in progress."

If the developers themselves don’t “ship” changes to any copy of their repository outside of their own privately accessible space, the (perceived from their perspective) problem is solved.

It is sensible; it is better to ship released versions in the package manager, instead of unreleased versions, at least by default. (If the package manager does not have the capability to distinguish in this way, then a user who wishes to use unreleased versions could compile it by themself instead.)

Unfortunately, some projects do not have any tagged releases (or, at least, doesn't have any yet), and might still be stable. I intend to add tagged releases to my "Free Hero Mesh" project eventually, in order to avoid this problem, that you can clearly have a released and tagged, with version numbers.

A distribution may need to patch bugs or other things in the software, to work with the distribution. This is OK, but they should probably mark this in some way, such as a nonstandard version number (e.g. "1.5.2.debian.1") or a different name. Possibly such nonstandard version numbers should also be included in the software itself if it has the capability to display its own version number, and not limited to the package manager. Sometimes there is a separate list of patches applied than the version number; this might also be usable (instead of or in addition to the version number).

Although "not ship work in progress" has many advantages, it interferes with "staying very close to HEAD of our dependencies" as discussed in the https://aboodman.medium.com/in-march-2011-i-drafted-an-artic... post. In other words, if your code is being consumed by another project that has extremely good test coverage, and your HEAD changes, then they can manage the risk of proceeding - even if they have no a priori idea of whether your latest commit is for a standalone improvement, or whether your latest commit is disruptive unless the entire work-in-progress is consumed together. They may find that managing this risk is easier than managing the risk of "huge chunks of new code suddenly showing up."

Solution seems to be features toggles, but then that means you have a somewhat complete product to begin with; and it being mature enough to support feature toggles.

Without concrete examples of good v bad behaviour it’s a lonely call in the void, IMHO. Without a clear commitment to solid versioning, where it is clear what is considered ready and stable v WIP it also doesn’t really help. Good will on all sides depends on understanding and communicating. This is a task for all.

I love those "under construction" images at the bottom. Takes me back to the 90's.

> when a project is being actively developed, tagged releases are the only safe option to ship to users.

Not always. https://github.com/clementine-player/Clementine is a great software, being developed but for some reason without release since 2016.

Last release does not work anymore, shipping master branch works.

The author isn't saying that it's the only possible way to ship software to users, he's saying that it's the only safe / good option.

Just because some project without a proper release cycle does it doesn't make this statement any less true.

Mostly agree. I recently caught up with Peter from Journey.io for an interview and he mentioned exactly this. Something along the lines of "the era of janky MVPs is over." There's certainly a balance of shipping an MVP and shipping crap. I think if you routinely ship crap, or things that are subpar, for the sake of speed, users will start to associate all of your work with crap.

I think maybe I saw it on Indie Hackers but there was a term I liked for this evolution in customer expectations called MAP, Most Awesome Product. While weirdly named, it’s the base product required for potential customers to go, “Wow, that’s awesome we need that.”

But then again, free and open source licenses enable everyone to do any modifications for any purpose whatsoever. That's the whole point. I would like it if the developers quit being patronizing towards people exercising their rights under those licenses. Yes, don't ship it, don't theme my app. We all heard you. Some people choose to not care, and that's okay.

Those same licenses also disclaim any warranty, so the buck stops with whomever applies random patches and takes money for it. In this case, the phone manufacturer shipping OS images. The phone manufacturer can duke it out with Manjaro, of course, and Manjaro folks can tell them to go pound sand and use Debian stable. This is well before upstream should even notice a shadow of kerfuffle happening.

The developers have come up with a multitude of ideas on how to be passive aggressive towards anyone either trying to contribute or submit an issue, stalebot and radio silence being only two of them, so I'm wondering why they just won't apply those techniques this time.

This is completely missing the forest for the trees. Just because something is not illegal doesn't mean it's a good idea. When a distro maintainer includes half baked patches for some third party software and the end user then has an issue with it, you can be sure they're gonna reach out to the software maintainers, not to the distro. By the time the back and forth helps everyone figure out that the problem is the version the distro packaged you've created a lot of useless noise and wasted plenty of time.

The problem is something like this: you develop packageA. A user of distroB is installing pacakgeA from distroB latest, and packageA is not working for them. distroB maintainers tell them to go ask packageA about the bug. So, the user comes and bothers packageA about this issue - even though packageA had no intention of distributing this in-progress version to users.

Now, of course, no one here is doing anything illegal. But, everything would be better for everyone if distroB, instead of taking packageA@master had taken packageA@1.0.1 or whatever the latest release is: better working software for distroB users, less support work (bug triage etc) for distroB, less work for packageA maintainers.

Since this is ultimately a social issue, I think an open letter seeking to convince the people involved to think about it and modify their behavior is the best way of going about improving this for everyone. Now, it may well be that the maintainers of distroB have valid reasons not to change their behavior and ignore this letter: all fine. Not saying we should tar and feather them, in any way shape or form. But if this is maybe a fixable problem, why not try to fix it?

If a user comes to me bothering me about some package some distro made of my software, I show the part that says NO WARRANTY and AS IS, and get on with my life.

Also, why should it “bother” anyone at all. You duke it out with whoever brought it to you.

It’s only a problem if you feel a need to please everyone knocking on your doors, which inevitably turns into burnout and you actually behaving harshly towards everyone in the end.

So your response to bug reports is to ignore them and tell the reporter to F off?

That should be common sense. Shrug.

I wonder if this could be solved with license terms.

Popular, OSI-approved licenses include clauses like "Neither the name of the <copyright holder> nor the names of its contributors may be used to endorse or promote products derived from this software" (BSD) or restrictions on the use of the original name (see Firefox/Iceweasel drama).

If you put in a clause like "You may not keep the software's name or support URL unless distributing an officially-released version", perhaps it would still be open-source as per OSI, and address those distribution issues? It's easy enough for distributors to patch in their own support mailing-list...

I think you could just assert that unmerged PR's have the following license:

This won't work with most licenses, if you don't own copyright over the totality of trunk. The PR is definitely a derivative of the trunk...

Would it fix the problem to develop with a more restrictive license, and only license the releases with a more permissive license?

Interesting thought :D

But it would probably create even more legal issues and also risks. A lot of people don’t use software, that has a non-standard way of licensing (like MIT, BSD, Apache, GPL). Just because there is a risk that you step into a legal trap.

If only the WIP commits were non-Free then relicensed to Free for each release, then the legal uncertainty would probably have the desired effect - no packager / distro worth their salt would touch any non-release with a barge pole. Such an unorthodox approach would risk scaring them off completely though.

In a roundabout way it's pretty much trying to re-invent the trademark system (ie. "don't distribute this code and call it MyProject without it being an authorised release"). Quite frankly far easier to just get a trademark if that's the desired effect.

> Quite frankly far easier to just get a trademark if that's the desired effect.

Trademarks cost serious upfront money - Germany alone is 290€ [1], EU-wide 850€ [2], the US 250$ [3] and worldwide is a hot mess [4]. Additionally, it seems like you need a lawyer in the US to handle the process if you're not an US resident, and you have to renew them every couple of years and not forget to tell the patent/trademark office of address changes. Not everyone is willing to put up that much money and effort for an open-source project that won't generate any income.

Additionally, trademarks usually force the holder to publicly register their name and address, which is simply not a good idea at all for many persons - trolls, spammers and outright criminals can and will use every bit of information they get to cause harm. And for collectives that run an open-source project, trademarks will be yet another issue - what to do when the holder of the trademark dies or disappears, or when they get into some sort of conflict?

[1] https://www.dpma.de/marken/

[2] https://euipo.europa.eu/ohimportal/en/fees-and-payments

[3] https://www.uspto.gov/trademarks/basics/how-much-does-it-cos...

[3] https://www.wipo.int/madrid/en/

I’m more concerned about changing the license of contributions during merge to master or a release. That something goes wrong there and the whole code base ends up „poisoned“ with non free code.

Software licenses need to work in a lot of different legislations, and it’s really hard to make a license that means the same thing in all countries.

Preventing distros from patching software at all would be a non-starter; there are lots of legitimate reasons to want/need to patch things.

Indeed, most patches I see in off-beat systems like nixpkgs, homebrew, etc, are not plucked from preexisting pull requests, but rather are fixes developed by the person who did the packaging and submitted upstream, then included as a patch until the first tagged release that has them merged.

Why would it fix the problem? That's besides the fact that this would complicate the legal situation of the project, including that of contributions.

Title is click-bait, it’s about a very, very narrow subset of the topic — that is Linux patches.

At least they let you know in the first paragraph but I agree even if it doesn't apply only to the Linux kernel.

If an article in Stamp Collectors Weekly has the headline "The Market is Getting Irrational," is it clickbait for that article to be about the stamp market?

It's weird to call someone's site about Linux patches clickbait because it's not about shipping half-finished furniture.

To me anything that intentionally misleads or overly generalize a topic which might be clearly and specifically addressed is click-bait by definition.

Author does not even mention that the post is specifically related to Pine64’s dependency on the Manjaro distro; a dependency that they not only self selected, but are funding. If they have such a major issue, solution is obvious, either change distros or fork it and only allow patches they are happy with into their ecosystem; not post a petition, of which so far only 16 people have signed since it was posted in June. Also, worth noting that Pine64 originally was built on Ubuntu, which has long-term releases, which is basically what author is asking for.

> In short: when a project is being actively developed, tagged releases are the only safe option to ship to users.

If this is such a problem that people need to be warned about it, why not just keep development on branches and make sure master is always stable?

That wouldn't help because they're not picking master to ship, they're picking mailing list patches and unmerged or rejected Gitlab merge requests directly.

If only they would ship master, that's at least somewhat sane.

The page is missing all these contexts. It don’t make much sense on its own

Reply on news.ycombinator.com