Why and How Java Developers Break APIs | Hacker News Reader

Why and How Java Developers Break APIs

arxiv.org

41 pointsecho-419 comments

As a relative newcomer, the most frustrating thing for me about settling into the Java ecosystem is the frequent breaking changes and inevitable dependency conflicts that result.

Breaking changes are sometimes a necessary evil. But they don't belong on minor releases. There should really be a stronger community standard that package maintainers sit on their hands and simply deprecate calls they want to get rid of until they can sweep them up in the next major release. It's also not cute to abuse semantic versioning by incrementing the major version number several times a year. (I'm looking at you, Guava.) That sort of stuff might be fine if a packages were relatively isolated affairs, but then the rest of the ecosystem doesn't really do anyone any favors by pulling in deep hierarchies of transitive dependencies that make it very difficult to limit one's exposure to the bad players. And then the build tools get to pile on and (as Maven does, at least) default to unpredictable dependency conflict resolution rules that obfuscate any latent problems.

I suppose it's no wonder that Java's the only place I've ever seen something like package relocation. But, at the same time, that practice has got to be the most distressing code smell I've encountered in my career to date.

Hopefully Java 9 makes some of this better (we're not ready to upgrade to it, yet), but I can't shake the feeling that this stuff really shouldn't have to have been such a hassle in the first place.

> "As a relative newcomer, the most frustrating thing for me about the Java ecosystem is the frequent breaking changes and inevitable dependency conflict problems that result."

I've got a couple of questions here because it's an area I find interesting.

Do you have a list of offenders (in addition to Guava) that you frequently get bitten by?

- What other ecosystems are you familiar with and how do they compare (favorably or otherwise) to Java?

- What in particular do you find unpredictable about Maven's dependency resolution?

I once wrote a static checker for Java. It found all breaking changes in APIs of all my dependencies. Any time functions were removed, parameters were removed, type signatures changed, it would catch them.

It would walk the maven dependency tree to find these, any time if found two versions of a library (say app depends on A and B, A depends on Guava 17, B depends on 18), it would log it as an resolvable issue.

On the project I tested it against, I remember it logging something like 250-1000 issues. I gave up. I'm not even sure where the code is now. Java is supposedly statically checked, but it lacks a real linker, as anyone who has ever seen a run time exception of "XXX is not a function" can tell you.

As far as Maven being "unpredictable", how it would resolve transitive dependency conflicts prior to 2.0 was undefined behaviour, and post 2.0 it is bizarre/lazy at best. It doesn't use the newest version, and doesn't flag major version conflicts. It just topologically sorts the graph and picks the highest dependency. The logic is bizarre. I have no idea if gradle handles it better, but they both face an unsolvable problem.

Shade is a pretty terrible work around. Auto-sandboxing conflicting dependencies in their own class loaders would be the most reasonable solution. As long as return values weren't passed between libraries.

Edit to add:

And this is not a theoretical problem either, as soon as you bring in a big dependency like Spring or Spark you are already 20+ levels deep in the tree, with dozens of these sorts of library conflicts, and hundreds of broken method signatures, just waiting to step on the wrong land-mine. (Not that I bet large ecosystem is any better)

I use maven-enforcer-plugin to force resolution of dependency conflicts. It's a pain at first to go through and add all the exclusions. But then I don't have dig through the dependencies in, e.g., a war, to find duplicate versions of a package.

Edit: Then I cross my fingers hoping there isn't a breakage or unexpected behavior somewhere.

Maven enforcer is a good idea for sure, but as I recall it just notifies you of a version issue, and allows you to explicitly select one or the other.

Sadly, these breaking changes can be mutually exclusive: e.g. a method was renamed that A & B both use, so no matter which version you pick something fails.

I remember once having to do a binary search through versions of Guava to find one between 15 and 18 that happened to be the one that was "just right".

How did your static checker distinguished between API and internals? How do you even do that automatically?

If I recall correctly, simply by checking only public methods, and assuming protected / private / package private were internals.

You got a lot of false positives then. Protected does not imply internal and public does not imply it is not internal.

The issue here is javailable packaging system, people need to do non-API public methods.

That was one of the reasons I've switched to Gradle. By default it uses the latest version, and you can even define the strategy for resolving dependencies.

> big dependency like Spring

a big offender here are the spring boot starter meta dependencies, which are a very opinionated set of mutually compatible dependencies, but God forbid that you want to use a newer version in your own project,

e.g elasticsearch 5 vs the "officially compatible" 2.4. welcome to jar hell

> - What other ecosystems are you familiar with and how do they compare (favorably or otherwise) to Java?

Opscode and many, many community cookbooks are a pleasant surprise there. Most cookbooks I've been working with have adhered to semantic versioning except for when a mistake occured. Backports of fixes are kinda accepted, but you have to do the work yourself.

I'd guess this is because someone writing a cookbook tends to be closer to operations and knows how useful a small, safe bugfix is.

> As a relative newcomer, the most frustrating thing for me about settling into the Java ecosystem is the frequent breaking changes and inevitable dependency conflicts that result.

After working with many languages, it feels like Java is the most stable ecosystem yet.

The JS ecosystem breaks APIs almost daily, similar in ruby (just look at factorygirl), and other such languages.

Maybe it’s better in Haskell or C, though.

I would say that C is comparable to Java in this regard, but Haskell breaks a lot as well. In Haskell, however, the community typically embraces breaking changes by standing up and fixing packages proactively; see, for example, Stackage and how they chase package maintainers once one of the package would break with a certain combination of other packages.

I have successfully compiled and used 30 year old Common Lisp code bases with minimal or no changes.

If there is any higher praise of that ecosystem, I haven't seen it.

I would recommend, as a general practice in any language, to never assume that version numbers signify anything other than chronological order. Apply the same level of caution and verification to upgrading a dependency from 5.1.1 to 5.1.2 as you would when upgrading from 5.1.1 to 6.0.0.

People are inconsistent with semantic versioning, so it can't be trusted. Even people who have been consistent with it in the past may behave differently in the future. I've been burned too many times and I won't even do minor version upgrades anymore unless I have 100% test coverage on the parts of my code that use the dependency.

That's why for internal projects we switched to using the build number for versioning. Developers rarely get SemVer right, or outright don't care about it. We've also seen patch versions breaking clients because of some transitive dependency upgrade which was impossible to forsee.

Ideally you shouldn't be changing dependency versions when you're just doing a patch, precisely because it's so hard to predict potential impact when you upgrade a dependency.

> Ideally you shouldn't be changing dependency versions when you're just doing a patch,

what if the patch included updating the (patch) version of a direct dependency? then you don't have much more control over what that direct dependency would bring on

Reply on news.ycombinator.com