Comment by alexhutcheson | Hacker News Reader

Comment by alexhutcheson | Hacker News Reader

Parent Full thread

1. You determine that a library used by a sizable fraction of the code in your entire org has a problem that’s critical to fix (maybe a security issue, or maybe the change could just save millions of dollars in compute resources, etc.), but the fix requires updating the use of that library in ~30 call sites spread across the codebases of ~10 different teams.

2. You create a PR that fixes the code and the problematic call sites in a single commit. It gets merged and you’re done.

In the multi-repo world, you need to instead:

1. Add conditional branching in your library so that it supports both the old behavior and new behavior. This could be an experiment flag, a new method DoSomethingV2, a new constructor arg, etc. Depending on how you do this, you might dramatically increase the number of call sites that need to be modified.

2. Either wait for all the problematic clients to update to the new version of your library, or create PRs to manually bump their version. Whoops - turns out a couple of them were on a very old version, and the upgrade is non-trivial. Now that’s your problem to resolve before you proceed.

3. Create PRs to modify the calling code in every repo that includes problematic calls, and follow up with 10 different reviewers to get them merged.

4. If you still have the stamina, go through steps 1-3 again to clean up the conditional logic you added to your library in step 1.

Basically, if code calls libraries that exist in different repos, then making backwards-incompatible changes to those libraries becomes extremely expensive. This is bad, because sometimes backwards-incompatible changes would have very high value.

If the numbers from my example were higher (e.g. 1000 call sites across 100 teams), then the library maintainer in a monorepo would probably still want to use a feature flag or similar to avoid trying to merge a commit that affects 1000 files in one go. However, the library maintainer’s job is still dramatically easier, because they don’t have to deal with 100 individual repos, and they don’t need to do anything to ensure that everyone is using the latest version of their library.

Your monorepo scenario makes the following unlikely assumptions:

1. A critical security/performance fix has no other recourse than breaking the interface compatibility of a library. Far more common scenario is this can be fixed in the implementation without BC breaks (otherwise systems like semver wouldn't make sense).

2. The person maintaining the library knows the codebases of 10 teams better than the those 10 teams, so that person can patch their projects better and faster than the actual teams.

As a library maintainer, you know the interface of your library. But that's merely the "how" on the other end of those 30 call sites. You don't know the "why". You can easily break their projects, despite your code compiles just fine. So that'd be reckless of an approach.

Also your multi-repo scenario is artificially contrived. No, you don't need conditional branching and all this nonsense.

In the common scenario, you just push a patch that maintains BC and tell the teams to update and that's it.

And if you do have BC breaks, then:

1. Push a major version with the BC breaks and the fix.

2. Push a patch version deprecating that release and telling developers to update.

That's it. You don't need all this nonsense you listed.

Semver is a second-rate coping mechanism for when better coordination systems don't exist.

Patching the code of 10 projects you don't maintain isn't an example of a "coordination system". It's an example of avoiding having one.

In multithreading this would be basically mutable shared state with no coordination. Every thread sees everything, and is free to mutate any of it at any point. Which as we all know is a best practice in multithreading /s

The same code can have multiple overlapping sets of maintainers. For example, one team can be responsible for business logic while another team can manage core abstractions shared by many product teams. Yet another team may be responsible for upgrading to newer toolchains and language features. They'll all want to touch the same code but make different, roughly orthogonal changes to it.

Semver provides just a few bits of information, not nearly enough to cover the whole gamut of shared and distributed responsibility.

The comparison with multithreading is not really valid, since monorepos typically linearize history.

Semver was enough for me to resolve very simply a scenario above that was presented as some kind of unsurmountable nightmare. So I think Semver is just fine. It's an example of a simple, well designed abstraction. Having "more bits" is not a virtue here.

I could have some comments on your "overlapping responsibilities" as well, but your description is too abstract and vague to address, so I'm pass on that. But you literally described the concept of library at one point. There's nothing overlapping about it.

I’ve lived both lives. It absolutely is an ordeal making changes across repos. The model you are highlighting opens up substantial risk that folks don’t update in a timely manner. What you are describing is basically just throwing code over the wall and hoping for the best.

We're still people capable of speech and communication, if something is that urgent, we can communicate it to our peers.

Changing code under their nose risks breaking bunch of projects. We can also fix this by rather communicating, right? But if we CAN communicate... then we can go back to the previous option (telling them to update) as it becomes just as viable.

Communicating is always essential, and can't be avoided.

> You create a PR that fixes the code and the problematic call sites in a single commit. It gets merged and you’re done.

What happens when you roll this out and partway through the rollout an old version talks to a new version? I thought you still needed backwards compat? I'm a student and I've never worked on a project with no-downtime deploys, so I'm interested in how this can be possible.

Version skew is only an issue when there's cross service communication. One service deployed on two different codepaths (in this case, using a different library implementation) is completely fine.

My example was a library, so both the library and it’s caller are in the same release artifact (binary/JAR/whatever).

If you’re changing the interface of an RPC service, then you can’t do that in a single commit, and need to fall back to something like the second approach, but with even more caution to make sure you properly account for releases and the possibility of rollbacks.

Reply on news.ycombinator.com