Depends – the missing linker for the JVM | Hacker News Reader

Depends – the missing linker for the JVM

github.com

124 pointsvoodootrucker29 comments

This seems less linking and more dependency resolution.

What was wrong with

> ./gradlew dependencies

and then sorting out your duplicate dependencies with improper version issues.

I have spent a lot of time down this road myself and most of it was not fully understanding the framework dependencies, munging of stackoverflow solutions, or apache-commons hell...sometimes a little bit of it all.

Not trying to crap on your hardwork but interested in your motivations if its something bigger which I am not seeing.

Long term I don't think you can properly suggest the appropriate dependency, this is why most build tools provide default selection and let you override with exclude patterns.

Additionally there is a linting tool that will parse through duplicates and offer to clean up your gradle files if that is interesting to you alongside your work.

Couple this to the fact that you can search for the class at

https://search.maven.org/#browse

to find the artifact hosting it, and it tends to be a couple minute job

Bonus tracks: gradle dependencyInsight is great for a similar task: finding out which artifact is bringing the dep.

That’s all fine and well if you have one or two broken dependencies from big popular libraries. This technique does not scale if you have a few dozen internal libraries and a few hundred internally deployed artifacts, or more. At that scale you need a more automated solution.

Anecdote: we have a large codebase with probably 100+ 3rd party dependencies, and dozens of internal modules, and that technique works pretty well.

Besides, where's the automated part on the OP solution? I agree the automation, either with this tool, or vanilla project management ones like gradle, could be beneficial.

I can’t speak to gradle, since we don’t use it, but none of the other tools I’ve touched look at ABI compatibility. It’s pretty easy to tell who brought in whatever version after it’s crashed, but catching that in advance is much much harder.

To the parent and GP, this is a problem in large code bases. As soon as you bring in a high level dependency like Spring, Spark, or DropWizard the graph goes 20+ levels deep and manual resolution becomes difficult.

Build tools do their best. Gradle has a reasonably sane version based solution. By statically analyzing the method invocations (and eventually properties too), this tool could (one day) just tell you the versions to use that would be most likely to work.

The long term vision would be to actually incorporate this as a dependency resolution strategy in Gradle so you don't have to think at all.

I guess its called linker to make the phrase sound like "the missing link for the JVM."

Maven equivalent for ./gradlew dependencies:

mvn dependency:tree -Dverbose

./mvnw dependency:tree -Dverbose (if you are using the Maven wrapper)

./gradlew dependencies will not tell you which versions play nice together.

Concrete example:

I dealt with a dependency conflict between two different packages that wanted different versions of Apache Commons. Both of them only claimed that they wanted "greater than or equal to X version." This wasn't actually true, though, because Apache Commons had removed some features and added some features, so the real requirements were

  Package A wants Commons >= X and < Y
  Package B Commons >= Z

where Z > Y.

Sorting that out meant manually doing exactly what this Depends package does - poking around with different versions and using the missing method exceptions and some other research to figure out the exact nature of the conflict, so that I could pin their versions and set up appropriate package relocation rules.

This tool would not just do that faster, but also more reliably - by doing it manually, I was only able to verify that I had resolved any issues with the code I actually exercised. A gap in test coverage would leave me exposed to run-time errors in production.

Nice!

I have two suggestions:

I'm probably too lazy to download and run a JAR but I'd probably run a Maven plugin from the command-line. Try packaging it as a plugin.

I often run into ClassNotFoundException when using an API but not providing an implementation, when using an optional feature that is provided in a separate dependency (Jersey HK2), and when using a feature that has been removed or broken into a separate dependency (JAXB). I believe these scenarios are all fairly common. Will your program help developers resolve those kinds of problems?

Thanks! This is great advice. I will look into it. I would definitely want it to be available for both Maven and Gradle, but I think this should be achievable.

Apache Gump [1] is a similar very old project, but meant to be more realtime.

[1] https://gump.apache.org/

That name makes me think of the adult diapers. Might have a branding issue

I think that is the joke.

If I recall correctly 'depends' is the name of a similar tool in Windows for debugging DLLs.

Given that this tool is for solving problems that stem from the Java platform's incontinent dependency management, that branding seems spot-on to me.

I can't imagine what sort of ill managed corp crapshoot frameworks you're importing in order to necessitate this tool :( but you might also try mvn dependency:analyze to help identify problem areas

Given a huge codebase: Spring. Blue-coloured j2ee containers. Old non-mavenized applications found as source jars, etc.

Spring has for some time solved this by using a bon with set versions of each library. Works great in Maven, and with Gradle via a plugin: https://docs.spring.io/spring-boot/docs/current/reference/ht...

Incredibly, there might be non-spring applications that need to be maintained, or even non spring-boot ones /s

I've heard some rumours about that, but wrote them off as too implausible.

That's not a linker, a linker produces an executable from object files.

There will always be a case where you can't solve the diamond dependency problem.

Two responses to the diamond problem:

- I see this often with Guava. A wants 18 and B wants 21, neither works with the other, but they both with with 20. Because this tool looks at the actual methods being invoked, even if 100% of the methods are present, it's fine as long as 100% of the ones being invoked are.

- Technically, as long as objects aren't exchanged between modules, it would be possible to use ClassLoader isolation to have two libraries each load their own incompatible dependency at run time. It's on my blue-sky TODO list.

Isn't choosing to sub in 20 just helping you build more on an unstable foundation?

1 - how can you be sure the method implementation is the same? That there aren't side effects you don't know about. (I don't think you can!)

2 - when it emerges that B2 really does need 25 what are you going to do? IMO it would be better to resolve this earlier than later.

> Technically, as long as objects aren't exchanged between modules, it would be possible to use ClassLoader isolation to have two libraries each load their own incompatible dependency at run time. It's on my blue-sky TODO list.

Not only technically, this also works practically. We've done this for years in a POS application in order to be able to load different incompatible versions of vendor-provided hardware drivers at the same time, with the decision on which to use (one, the other, or maybe even both at the same time) being done at runtime. It's based on OSGi for modularization, which also provides other benefits besides classloader isolation, like allowing to bundle multiple native libs for different OSes and bit-nesses within a single jar and automatically choosing the right one to dynamically link into your JVM process that is needed for a particular platform that the application finds itself running on.

Taken to a logical extent, I think this tool could be used to generate conflict-aware ClassLoaders at application start time, and automatically give each module the version it wants.

I guess it would take generating a new jar with a bootloader main() to wrap the existing main()?

Great stuff. Never got to building this myself but this is awesome.

Reply on news.ycombinator.com