Did I break you? Reverse dependency verification | Hacker News Reader

Did I break you? Reverse dependency verification

developers.soundcloud.com

73 pointsr4um69 comments

This might be unpopular, but I don't think libraries should ever break backwards compatibility, at least not with enough regularity to require all of this. Sure if there is a major security thing, or something outside of your control then you might have to. But if you just want to change a function's signature, or return value, because you think it will help "clean things up" could you please just make a new function, or make it work both ways if your language allows it. I understand the desire to tear out a messy bit of code and make it beautiful, but it always leads to a bunch of headaches for little gain.

There's no way to entirely avoid breaking compatibility. If you allow small API breakages then at least all of the team is aware of the possibility and designs their processes to handle backwards incompatible changes. If you design your processes assuming backwards compatibility then you'll run into the 1-in-1000 case where it's not possible and have to stop everything to handle the major update.

> There's no way to entirely avoid breaking compatibility.

Why?

Because you've got no idea how your code's being used in the wild an one person's bug is an other's feature. So any observable change in behaviour is a possibly breaking change.

And then people go and muck with your internals even if you protected them, because that's the only way they found to do what they needed to do, and they were much more interested in doing the thing than following your guidelines.

And that's before even mentioning broken features which requires changes which will invariably break other stuff.

The only way to know for sure you're never breaking compatibility is to never release any update. Otherwise you're playing the odds.

https://www.hyrumslaw.com/

Also more facetiously, xkcd/1172

That might be a real issue for people which strive for a high level of compatibility, such as engineers at Google, but I don't think it applies to less than 0.1% of compatibility-breaking changes. Also, such things are fixable if they are important - just provide the old behavior by default and add a flag or keyword argument which selects the updated behavior. And it also helps to make clear that the documented behavior is what you promise, nothing else.

I think API breakages are usually just caused by developers which failed to define consistent and thought-out interfaces from the start, and library developers which do not want to do any extra work to continue to provide APIs with the old semantics.

I know that it can be made to work, because I have worked in industrial automation and PLC libraries, and while there are not always and everywhere geniuses at work, in this field legacy interfaces are just never scrapped because they need to continue to work. What you publish needs to continue to work for 20+ years, because nobody wants to scrap a six-figure printing machine just because of an update to the PLC libraries. So, it does not require geniuses to do that. It is more a matter of what the consumers of a library tolerate.

And the same is valid for language evolution. Common Lisp for example, or C are extremely stable. Python users have come to tolerate a far higher amount of breakage and instability, but I do not buy that this is in any way technically required. The breakage might not matter that much if the code is only used in fast-moving start-ups of which 99% will be bust anyway in four years time. But there are domains where the costs of this are just too high, such as science projects, industrial automation, enterprise systems, and many more, because systems are too complex and expensive to update.

Also, Hyrums law are about unforeseen breakage because people did not relay only on the documented API specification. This is undesirable, and in fact hard to avoid completely.

But most breaking changes such as in semantic versioning major number changes do intentionally break client code, which is an entirely different thing. There are a few rules one can follow which make it pretty improbable that such foreseeable breakage will happen. And this will constitute more than 99.9% of API breakage - even more if you stipulate that undocumented features must not be relied upon, and client developers follow that rule. Developers relying on undocumented behavior is usually due to a lack of clear documentation, so it is largely avoidable, too.

Something that here a lot here is having comprehensive, trustworthy documentation.

This gives you the moral high ground to adopt semantic versioning: the correct behavior is the documented behavior. If someone has code that relies on an undocumented method or undocumented behavior, it's fair game to break that with only a minor version bump.

If your documentation isn't any good, SemVer is almost a fiction.

I second that. Also, semantic versioning requires documentation to give specific meaning to an interface.

Are you saying that it is better to have to deal with a bunch of tiny changes, so that when you have to deal with a big change you'll be used to it? If so, I strongly disagree.

I would much rather have to deal with a big annoying change every 10 years, than a small annoying change every 6 months (which would still likely need the big one every 10 years).

/orders

/orders2

/orders3

/orders4

/neworders

/getordersbutonlythistype

If you're changing the endpoint that same amount but keeping the URL, you've just imposed a ton of churn on everyone downstream.

It's a tradeoff, like everything. Try to get the API right the first time. Try to avoid introducing new APIs. Try to avoid breaking the old API. There's no silver bullet.

After all, your business never changes and neither do the constraints.

Who are you arguing with?

And /orders uses an old data layer in a way that is incompatible with the new one, so the team can't deprecate the internal DB abstractions. And they can't introduce a new caching layer without making it work across the new and old abstractions. Etc, etc.

I agree with you. Churn for churn's sake to make things marginally better. One man's refactoring is another man's refucktoring.

Depending on a library could also insert a record on the library's CI system to indicate the dependency. The library CI will ping a URL with a version number to indicate that "you need to re-test with this version". It would certainly make software more robust and evolvable.

I've thought about it before and called it library mesh: https://github.com/samsquire/ideas2#10-library-mesh

People don't understand how their code can mess up others in unexpected ways. A few weeks ago my youtube just stopped working. Turns out, a change made to the apollo-graphql extension caused window.process to be readonly and youtube's code was not expecting that: https://github.com/apollographql/apollo-client-devtools/issu...

I know this isn't a dependency issue, but it's a great example of why things like semver don't work perfectly in practice. It's hard to figure out what the effects of your code really are in the end. You might assume a change is minor when it really is major. I wish programming languages could figure out what the version change should be for a given code change.

Sure, but that's just a bug in an application introduced by accident, and fixed when they found out about it. That is going to happen no matter what. The same thing goes for libraries. That is the best part of this tool, since they control the library and all the clients they can automatically run their tests across everything. Which would help you catch more of these types of problems, but it will never be perfect.

My point is that if a user of your library is failing after a change, you should treat that as a bug in the library. Backwards incompatible changes should only be on purpose, and only when absolutely necessary.

Of course, it's only their own code they're affecting here, so it doesn't really matter as much as a library for users you don't know. Plus it is a pretty cool solution, I just don't think it should be necessary.

"Don't change the function signature" and semver are things that are absolutely hammered into devs who work on library code. I'd be hard pressed to find somebody even tangentially involved in software development who doesn't understand this.

Semver doesn't cover changes in behavior. What a "breaking change" is when it comes to behavior changes is up to the person who implemented it.

>Semver doesn't cover changes in behavior.

It does:

https://semver.org/

"Software using Semantic Versioning MUST declare a public API. This API could be declared in the code itself or exist strictly in documentation. However it is done, it SHOULD be precise and comprehensive."

If the API changes semantics between different releases, it is a breaking change. It does not matter whether the types in the function signature is identical. If

    printReport(int invoice_number)

prints a report in version 1.0, and deletes the database or launches nuclear weapons in version 1.1, it does not follow semantic versioning.

But the core problem with semantic versioning is that it somehow legitimizes to make breaking changes all the time. It is not OK to beat somebody up in a bar. It is also not OK to tell somebody that he is going to be beaten up, and then beat him up.

Now, semantic versioning holds the idea that it is OK to make breaking changes, if only you tell people before. And this is a lousy idea. Breaking changes break other people's code, especially if they use it in deep graphs of dependencies. And they can't necessarily do anything about it, because they might have two dependencies which are depending on your library, only that the first one now requires version 2.0 at least, while the second one will not work with anything newer than version 1.99.17. And thus the whole project is broken. Actually, requiring the new version 2.0 in the first dependency is a breaking change, too, because it breaks backwards-compatibility.

And this is something which the Python devs just did not comprehend - that this kind of breakage trickles up dependency graphs and makes it extremely difficult for users with a limited budget to make upgrades, even if most of their features are highly desired.

I've come to the same conclusion. Infrastructure and libraries breaking backward compatibility should not happen, it is not a sign of technical quality. It is almost always possible to provide wrapper interfaces which contain the old behavior and function signature as a special case. And if that does not work, it is better to release a different library.

There is also one interesting insight I had when looking at error codes and exceptions: Normally, backwards compatibility is broken when you take things away from an interface - functions from a library, values from an enumeration type, and so on (Rich Hickey has made a brilliant talk about this). However, the general thing is that backward-compatible change must never narrow pre-conditions, nor widen post-conditions.

This means that it is not OK to just add extra error codes to the return values of a function interface, or to add new classes of exceptions. This breaks backward compatibility.

You can of course add enumeration values, optional arguments, keyword arguments and such to function call arguments, and these are great ways to make interface changes backwards-compatible.

Another insight is that backward-incompatible changes tend to cascade up larger dependency graphs. The broader and deeper the dependency graphs becomes, the more probable it is that a backward-compatible change (e.g. requiring a newer dependency) becomes itself a breaking change up the dependency chain. The philosophy in this age seems to be that of coourse the other library authors should "just fix their stuff", they get the responsibility of upgrading to the new version of the dependency. I do not agree to this at all - I think if something changes in a component, and this breaks the software, the responsibility is always within that component, and nowhere else. If some change in the component causes breakage, this is a breaking change, by definition. And this is also true if it is an upgrade from Qt4 to Qt5, or OpenCV, or imageio-py2 to imageio-py3 or such.

One could increment the mayor version number but this does not fix the problem. What fixes the problem is to not break stuff in the first place.

I am quite convinced that as in the future, ever more programming will be done in libraries and components, there will be ecosystems which do not accept that kind of breakage. Linux is already a good example, and it is wildly successful.

I think widening post conditions may have to be possible in some cases. In my experience, programs should only depend on three possible error states: ok, eagain, and everything else. I guess it's fine for library authors to promise more, but I'd only do so in the case where I think that calling code can respond with more specificity to other conditions, which I think is pretty rare.

Might in some cases be OK, but it is also how you get "general protection fault" and rampant uncaught exceptions in some earlier versions of Windows.

In the protobuf library, there are some protections against cases like this. For example, they add extra enum values that are explicitly labeled as "do not use" to prevent switch statements from assuming that an enum is complete and future-proof. https://github.com/protocolbuffers/protobuf/blob/master/src/...

> And if that does not work, it is better to release a different library

Is that really a solution though? You're just going to end up with an unsupported old version and a supported yet breaking new version which probably doesn't even have upgrade guidance... Not sure I see it. I think libraries should work hard as you say to not break backwards compat, but I also think the library author has the right to change some function signatures when they change major behaviors too

You can just fold the old interface with the old semantic into a special case of the new interface, and provide a wrapper function with the old signature. This avoids code duplication.

> I also think the library author has the right to change some function signatures when they change major behaviors too

When the the semantics change, it is already a backwards-incompatible change anyway, so it is cleaner to provide an extra function name for that new version.

There might be extreme cases where it is not possible or desirable to keep backwards compatibility, such as hardware bugs in Intel CPUs speculative execution, or APIs which turn out to be so thoroughly botched that they cannot be made safe. But these are exceptions.

One thing is to compose from backwards-compatible components makes it much much smother and easier to upgrade to new versions in a system which has many components. Paradoxically, it makes upgrading easier, and this is what library authors ultimately really want from users. While breaking compatibility does not only leads to a rat-tail of consequential breakage and forced upgrades which leads to more breakage, and makes everyone hesitant to do even small upgrades, ensuring backwards-compatibility leads to people upgrading without fear that they are interrupted for weeks and months.

At big companies this attitude eventually yields a stagnant product. No one is confidant that any impactful change wouldn’t break something and no one knows if that something even matters to customers. Even worse there may be teams that will make 0 changes as they’ve been marked as KTLO.

Having tooling to acknowledge that in fact no one will be affected by a change is a huge productivity boost.

Having tooling to acknowledge that in fact no one will be affected by a change is a huge productivity boost.

Yes I do agree with that part

Years ago I was blown away when I saw a presentation by a Google Java library developer. He was able to make changes to his library, then automatically build every Java application throughout the enterprise which used that library to check for automated test failures. This allowed the library team to move fast with confidence.

And you can too! http://Bazel.build

KTLO = Keeping the lights on, for anyone else who hasn't heard the acronym before.

> At big companies this attitude eventually yields a stagnant product.

Such as Linux?

Expecting library authors to get their design entirely correct the first time round isn't reasonable. When libraries make backwards-incompatible changes it's usually because the original design was flawed in a way that made continuing to support it, while adding new features, infeasible.

You're welcome to evaluate and select libraries with the best possible track record for avoiding backwards incompatible changes - I do that myself. But, especially for libraries that are relatively early in their development process, having a hard "no breakages" policy isn't realistic.

I still think semver is the best fix for this. If you make a backwards-incompatible change, you bump the major version number and you extensively document the change.

If you're consuming libraries, the best way to address this problem is with solid test coverage. That way any time there's a new library version you can lean on your test suite to help you understand if it's going to break anything for you or not.

> Expecting library authors to get their design entirely correct the first time round isn't reasonable.

People shouldn't be publishing such libraries, nor should people be using them. With the exception of my first few FOSS projects, all of my open source libraries represent the 2nd, 3rd, or even 4th incarnation of a concept.

That's just being responsible--don't knowingly litter the ecosystem with junk.

I take a similar view at work. People are too quick to break out components into modules and libraries, long before they have a firm notion of what a proper interface should look like. If you can't commit to a stable API, then often times it's better to just copy whole functions and files between projects and let them evolve naturally. It takes time to see where that evolution goes, and which commonalities emerge in the implementation and surrounding application.

But what about sharing bug and security fixes, you ask? Well, without a stable API you'll invariably end up with different projects stuck using different versions, anyhow. That's why people invented Docker :( What you end up with isn't an actually shared library, just the pretense of sharing, plus a bunch of unnecessary build complexity.

> People shouldn't be publishing such libraries, nor should people be using them.

That's why we have pre-1.0 version numbers (and alphas and betas) - they let us release code early for people to provide feedback before we've committed to a non-breaking API.

Even with a stable 1.0 release things change. People using the library may come up with logical feature requests that the original design didn't consider, and which require backwards incompatible changes. That's when you bump to a 2.0 release.

My dream project would have a version number of something like 1.237 - meaning there have been 237 new features releases since the 1.0 release without backwards compatibility breaks. My hat is off to any project that pulls that off though, it's a rare accomplishment!

If the new feature is incompatible with the old library, it's a different library. It should get a different name. If you're all out of names, just append a 2, as in jinja2 or psycopg2. No need to break existing software or to make it impossible to install the new library alongside the existing library.

That may not be necessary, depending on how everything works. When I use a v2 of a Go library, it doesn’t prevent me from also using v1 somewhere else in the same project.

GP is a python dev, so I'm talking python to him.

You're basically suggesting reimplementing semver in a clunkier way. Just bump the major version but continue supporting the old major version with security releases etc.

This is a solved problem, it's just that some library maintainers don't practice proper communication of breaking changes.

If your breaking changes are a total paradigm shift in the intended usage of the library/API then sure, it should be a new library. But quite often those breaking changes are just things like "field Foo is now required when performing process Bar" in order to prevent some sort of invalid state from being reached. In those cases communicating the breaking changes via versioning is the way to go imo.

> You're basically suggesting reimplementing semver in a clunkier way. Just bump the major version but continue supporting the old major version with security releases etc.

> This is a solved problem, it's just that some library maintainers don't practice proper communication of breaking changes.

It's not just a communication problem, it's also the way that a lot of language tooling works. For example on the JVM you can (mostly) only have a single version of a given (fully qualified) class on a given application's classpath, so it's quite important for incompatible new versions of a library to use a completely new set of class names - thus you'll see e.g. commons-lang3 using org.apache.commons.lang3 alongside commons-lang using org.apache.commons.lang.

Just introduce a new function. Don't break old users.

> You're basically suggesting reimplementing semver in a clunkier way.

There is a huge difference: Most languages and runtimes do not support having the same dynamic library twice in the same process. This means that if you just bump the major version number, you can only have one version of a library in a process. Now consider the case that your library L1 has a dependency L2 and is included in an application A which also includes another library L3 which has the same dependency L2 as you.

Now think what happens if L2 has a breaking change, and you upgrade L1 to use the new version, but L3 does not work with the new version, and does not upgrade it. In this case, the application A - the user of your library - is broken, and there is little the developer of A can do about it.

However, there are two kinds of libraries: One which access global objects, like GUI libraries, the glibc socket library, and such. Or which provide data interchange formats, such as Python's Numpy. These regularly need to be unique across an implementation.

And the other ones which do things like data transformation, parsing configuration, some numeric processing, and such. In this case, you could link L2 statically, if you were using C. r you could L2v2 and the library l3 could use L2v1. But in most languages and runtimes, this will only work if L2 version 1 and the incompatible L2 version 2 have a different name.

> Just bump the major version but continue supporting the old major version with security releases etc.

My experience is that the "continue supporting the old major version with security releases etc" almost never happens. People release incompatible versions and force upgrades on users because they do not want to do extra work. If they were supporting older versions, they would still have to do some of the extra work. They might do it some time in some cases but even then they will cease to do it soon.

If one wants continuous updates and support, one is usually MUCH better off to use libraries which just do not do compatibility-breaking changes. Because these have decided to do the work required up-front, and have little extra work with supplying urgent fixes to an old API interface.

Also, it is easy and much less disruptive to declare new features (and not the whole API) as unstable while they still may change, and promise stability for the rest.

Getting a design right the first time is hard, getting a design right without any users is really hard.

Also look into how many massive libraries basically started as someones weekend project for fun. I posit that the quality of the open source ecosystem would go down, not up if we tried to somehow enforce these kinds of standards on public projects.

Is Linux unreasonable? The kernel has been backwards-compatible to user-land from the start.

Or TeX, C, RS232?

Same with x86. Even modern x86 CPUs boots in real mode so you can run a baremetal 8086 binary from the 1980s.

> The kernel has been backwards-compatible to user-land from the start

Maybe at the source level ... at the binary driver level there is no stability at all, and it's a huge reason why the driver scene for linux still languishes in so many areas.

Now there are reasons linux has chosen this model but it nonetheless demonstrates this is not a "free" decision without any consequences.

> at the binary driver level there is no stability at all

You are evading the argument, what we were talking about was stability of APIs in respect to the userland code. Linux never promised binary compatibility at driver level.

Also, Linux has far better stability in hardware support than competing systems. If a relative tells me they have a photographic slide scanner or printer that stopped to work with some Windows x.y release, chances are quite good that it is continued to be supported by Linux.

I didn't say it made the projects themselves unreasonable, I said that projecting those expectations on other projects is unreasonable.

Those are all inspiring examples of projects that went truly above and beyond the norm and achieved great success as a result.

I don't know about RS232, but Linux, TeX and C all ended up really inelegant and hard to work with. Maybe the backward compatibility is worth it in your case; personally I'm far happier using something that has been ruthlessly refactored to keep everything clean even if it means I have to update things on my side sometimes.

> When libraries make backwards-incompatible changes it's usually because the original design was flawed in a way that made continuing to support it, while adding new features, infeasible.

I think that "infeasible" is rarely true.

Let's take, as a well-known example, Python3. Well, the python developers wanted to change the default string representation, the fact that "print" was a statement, and the behavior of integer division, but this absolutely could have been done by things such as a new syntax for Unicode strings (such as, huh, u"hello"), making integer division dependent on "import from __future__"statements, using a *.py3 file extension for breaking new syntax, and so on. It would have been more work for developers, true, and this was ultimately their decision, but it was not technically required.

What developers of library and infrastructure need to realize is that very soon there is far more code which depends on established APIs and that breaking this code is, on absolute terms, more work, and is disruptive in collective terms.

It is like a tree where most of the biomass is in the leaves, not in the trunk. For example, there is a Wiki software named MoinMoin Wiki, written in Python, and supporting multiple markup formats, which is written in Python. It is popular and also used for the Python wiki, so you could be assured it is used in infrastruccture. But there is no Python 3 port because the developers did not had time for that, and in this respect the breaking changes of Python 3 are disruptive. There are also many scientific code bases written in Python 2 which will never be ported to Python 3, because there is simply no funding and no scientific career for such work.

And this becomes very visible in larger projects which depend on a lot of libraries which in turn have transitive dependencies on multiple levels. Just look at the full dependency graph of something like Gazebo, Tensorflow or Gimp. The former might well have several hundred dependencies As soon as one of there libraries demands that a newer version of the dependency is used, and this dependency introduces breaking change, it is likely that another dependency in the same project is broken by it. And telling to users "to just fix and upgrade" is not going to work, it does not scale, and people will at some point stop to upgrade your stuff altogether. Imagine the Linux kernel worked this way! No sane person would use it.

Semver is one way, but yes.

It annoys me when libraries "clean up" (churn) their codebase when they should've thought more carefully about designing the interface (the contract) before publishing it. Code churn often creates more pointless work for N library users. I swear that most code churn is to make a project look "alive" or to give people something to do / job security.

Deprecate slowly and have semi-automated user code fixups and notes.

I remember reading an article on MSDN(a developer info system - I call it a system because it was also available on CDs as internet was not available everywhere) by a Microsoft employee. The article was titled some what on the lines of "How the marketing teams won over the engineering teams!" The article mentioned a tussle within Microsoft between the marketing and the engineering teams. The engineers were pushing for providing backwards compatibility which slowed them down and took time in providing new things/features that were the next sought after tech. The marketing teams on the otherhand were very frustrated that new sought after tech was not coming out faster so that they can then market it to the world. That tugh-off war was ultimately won by the marketing teams at Microsoft as per the article.

Reminds me of this article:

https://www.joelonsoftware.com/2004/06/13/how-microsoft-lost...

I actually genuinely think library authors should choose whatever trade-offs they want, and consumers should choose between those libraries based on the trade-offs they're comfortable accepting.

The "always backwards compatibility" and "never backwards compatibility" schools of thought might just as well be "no programmer should ever use a garbage collector" or "all programmers should only ever use garbage collectors".

Tools like OP help people who want to accept backwards incompatibility.

I don't do modern coding with "bunch of dependencies downloaded at compile time" so now I am having fears about being "like so out of touch man". Just reply and tell me I am out of touch if I am so far off base here.

If the updated library breaks your stuff then can't you just... not update?

And if you want to update to the new library for XYZ new cool stuff it does, then you are already putting in effort to utilise XYZ. In which case you can sort out the new ABC fix too.

It is extra work but it is part of having updated to the new coolness.

For other projects doing something similar, the Rust project has Crater (https://crater.rust-lang.org), which I believe is automatically run on betas, and can be run explicitly on PRs. But is not by default. Because it builds tens of thousands of crates using the specified compiler, and takes 3 days to complete a run.

Crater doesn't just run builds! It also compiles and runs unit tests to look for regressions between compiler versions. When I last looked the tool grabbed every library on crates.io and every repository on GitHub with Rust code, but they may have cut down the scope slightly as the ecosystem has grown

This is a good technique, and it's preferable to investing a bunch of effort into building a comprehensive suite of unit tests. But this technique is applicable to more than just core libraries; you can apply this to your entire stack, and to your services: Test that you don't break your reverse dependencies, by running those reverse dependencies, like in http://catern.com/usertests.html

Following reverse dependencies on github (trivial with the search api for enough cases to make a difference) is something that I very very strongly wish that Pypa had done for the pip 20 resolver change. It was even suggested to them by several people, but the suggestions were dismissed because breaking countless CI toolchains across the globe was considered fine because the new resolution behavior was "correct".

Instead we got https://github.com/pypa/pip/issues/8713 back in August and then https://github.com/pypa/pip/issues/9187 in November which is pinned and still open to this day.

Ok, if I understand this properly:

They have a core library that other projects use.

They want to release a new version of this core library, and upgrade all the projects to use the newer version.

If the CI breaks for the projects using the library, they want to catch it.

This still depends on the projects using the library having enough and good enough testing for something to break in the CI when this core library does something bad. That is a huge investment (having those high quality tests). The rest of this is just a script that runs the tests from those projects when people merge code into the core library, which seems like very much not a big deal.

The former approach was release + upgrade all projects. Now if one of those fails post-update you have to build a new release and upgrade all projects _again_. Multiply that by hundreds of services and it can become pretty annoying.

By pulling all the reverse dependencies and testing those _before_ releasing & upgrading you reduce the chance that you need another bugfix release.

And with hundreds of projects the average quality of tests does not matter that much. Some projects will happily pass, while some may fail. Those that fail will provide valuable insights.

> The former approach was release + upgrade all projects. Now if one of those fails post-update you have to build a new release and upgrade all projects _again_. Multiply that by hundreds of services and it can become pretty annoying.

I think there are actually at least 3 possibilities when you find a new version of a component breaks something that depends on it?

  1) Continue using the old version for however long it still meets it's requirements and then do (2).

  2) Make the local changes required to be compatible with the new version.

  3) There is actually a bug in the new version and you need to fix the dependency.

If (3) is happening a lot it suggests that your test suite isn't good enough or you have some design problems that are making it impossible to provide a stable API.

What it sounds like is that they don't have many unit tests for jvmkit. So rather than write extensive unit tests, they're using integration testing with all the dependencies. I guess it makes sense, network stacks are hard to unit test.

Yes, and the whole solution allows you to plan / decide on (2) and (3) - both need that awareness.

And (1) is common in the open source and cross organization world. But it is less desirable if both the consumers and producers of the library work in the same place. Version spread causes maintenance cost. And at one point it was a common theme that "an upgrade of JVMKit would have avoided this incident".

Skimming this, because I'm in a meeting, but it reminds me of contract testing, which I'm working on implementing at work today with Pact.

Nice - I've been thinking stuff should work like this for a decade+

Reply on news.ycombinator.com