Hacker News Reader

Comment by mattfenwick | Hacker News Reader

Sorry, I think I'm missing something because I don't see the problem -- why can't foo and bar just use different versions of baz? Is it not possible to do that?

View on HN

Let's say your app my_app uses foo and bar. They both use baz. You give them each their own version of baz: foo gets baz 1.0 and bar gets baz 2.0.

Now, at runtime:

    my_app calls foo.giveMeAWidget()
    foo calls new baz.Widget() on its 1.0.0 version of baz
    my_app gets that back
    then it calls bar.doSomethingWithAWidget(widget) and passes it in
    bar calls baz.flummoxAWidget(widget) and passes the widget in
    baz.flummoxAWidget() starts doing stuff with the widget

The last step is bad. You have baz 2.0.0 code that assumes it has a baz 2.0.0 widget, but it's actually a 1.0.0 one. It could work, crash, or fail in some subtle way.

Note that this isn't statically detectable. It's based on how objects flow around in memory at runtime.

Now, in cases where you don't pass objects around like this, you'll avoid this problem. But it's really hard to tell when that's the case and when it isn't.

"Doctor, it hurts when I do this."

What you're describing is an inappropriate intimacy antipattern.

Because people keep pinging me on twitter about this, I feel like I ought to explain a bit more.

What's going on in the pattern described is that you're getting a `BazWidget1` object from the `foo` module. Then, you're passing that object you got from `foo` into `bar`, which passes it to `Baz2Flummox(BazWidget2 widget)`.

Why are you flummoxing widgets that you don't know the origin of?

What kind of program needs to mutate state on a object, by passing it from one object-mutator to another? Why do you not have clearly defined ownership of objects, and clearly defined functions that take arguments and return values?

"Inappropriate Intimacy" is a code smell where one class is delving into the inner workings of another, depending on things that ought to be private. The `BazWidget` should be a private implementation detail of `foo` and `bar`, but instead, we are passing this implementation detail from one function call to another.

More specifically, we have `BazWidget1` and `BazWidget2` objects being used interchangeably. It's tempting to blame this on the package manager or module system, but it is just a badly designed program.

This is one class of errors that a "strongly" typed language can often detect and prevent at design time. However, I've seen C++ and Java programs with the same problematic antipattern, just with more layers of Interface wrapping. And, whatever, JavaScript is what it is, and is not "strongly" typed. It lets you pass any value as any argument to any function, and leaves it up to the callee to decide what to do with it.

Personally, I have no strongly held opinion about who's best to handle this responsibility: the compiler, the caller, or the callee. There are benefits to "loosely" typed languages as well, and I'd rather not make this about that.

I've referred to this sort of thing as a "gumby baton". You're passing an object from one worker to another, and each one mutates it a little bit, like runners in a relay race where the baton is made of clay, so it gets the finger print of each worker in turn.

This is a terrible antipattern! This is how we end up with middleware depending on other middleware having been executed in exactly the right order. It is terrible for reasoning about program behavior, and results in unexpected behavior when workers are combined in novel ways. Making programs harder to reason about makes security virtually impossible, and increases the cost of maintenance and re-use. Even up front, it is a challenging pattern to use in building an application, though it sounds appealing in principle if you've never been handed a warped and mangled baton.

So, when I say "Doc, it hurts when I do this", I'm implying that the proper response is "Don't do that".

Ultimately, it's not the compiler's fault. Gumby batons exist in C++ and Java and C and are even possible in pure functional languages. Be on the lookout for it. The compiler won't protect you. The module system won't protect you. You have to use your human brain.

Another caveat just to avoid any "tu quoque" responses: I've made this mistake (and sworn to never do it again!) many times. The most egregious offender in Node is mutating the req and res objects. But, it can be very subtle and hard to spot in the initial design. We just fixed a bunch of really subtle bugs in the lockfile module by changing how it was handling the options object, because it had taken on a gumby-baton behavior internally.

Mutation isn't necessary to demonstrate the problem. Consider three libraries: first, there's a basic, widely-used datetime library. There's also a timezone library depending on datetime 1.0 and a dateformat library depending on datetime 2.0.

  # Get the current time in the PST time zone (returns a 1.0 object)
  now = timezone.now_in_zone("PST")
  # Format the date for display (accepts a 1.0 object but depends on 2.0)
  formatted = dateformat.format(now)

Now the problem: imagine that datetime 2.0 switched from one-indexed months (1 is January) to zero-indexed months (1 is February). The timezone library depends on datetime 1.0, so it used "1" to indicate January, giving me a datetime value with month=1. The dateformat library depends on datetime 2.0, so it incorrectly interprets that month=1 as February. All of my January dates will now be incorrectly formatted as February.

(Switching month representation is a rather drastic example, but it's also clear. Substitute a more subtle data format change if you'd like.)

There's no mutation here, and the dependency graph is trivial. I just received a datetime from one library and passed it to another library.

It's possible that I'm missing something, but I've been asking Node users about this for a couple years and I usually get blank stares. I also have no horse in this particular race. Vendoring seems like a great idea to me, but I fear the uncertainty of this version mismatch situation.

The problem is here: "(accepts a 1.0 object but depends on 2.0)"

This case actually has nothing to do with nested dependencies. Your time zone lib is returning a datum with one type, which you're passing to a formatting lib that expects a datum with another type. This can happen if your libs have dependencies that are totally different libraries instead of the same library with different versions. It can also happen if your libs have no dependencies at all. This is not an issue of dependencies but of you not understanding your libs' APIs.

In most languages, a type mismatch would always correspond directly to a type name mismatch. In e.g. Python (since it has a clear module system), if I know that f() returns a datetime.datetime, and I know that g(t) takes a datetime.datetime, then I know that they will compose. (Modulo bugs of other types, of course.)

When using NPM, I don't have that guarantee. The docs for these libraries could clearly state that they'll integrate around the datetime type, but that may be false in practice. And I'll only know that ahead of time if (1) I know that they both use datetime internally, (2) I know exactly the versions of datetime that they use, and (3) I know exactly how datetime changed in between those versions. With non-vendoring package management, I don't have to know any of these things.

You've correctly identified a guarantee that you don't have with npm. In practice, anecdotally, I've never run into this issue, while I have many times and with much pain dealt with conflicting deep dependencies using bower and bundler. Which may explain the blank stares. It's a tradeoff I am personally happy to make.

> clearly defined functions that take arguments and return values?

Sure, but what are the types of those arguments and return values?

If you call what I describe an anti-pattern, you're basically saying that packages can only interact using primitive types defined in the language. You can reuse code, but not data structures.

I think that's too much of a limitation. I want to reuse code that defines matrices, and vectors, and interesting collections, and business model objects like mailing addresses and currencies. I want to make games that use a mesh type defined in one package and pass it to a collision engine in another.

Saying "you can't use any user-defined type in any public API" is an incredibly harsh limitation, and what do you get for that in return? The ability to bloat your application with multiple versions of the same library?

In this situation, "foo" or "bar" or both should not depend on baz, but have it as a peer dependency. There is nothing broken, just a very specific case not so easily identifiable, but very easily fixable.

Sadly, this isn't really a solvable problem. On one extreme you force all transitive dependencies to use the same version. On the other extreme, you bundle n different versions of a library. Organizations such as Google have strict policies to enforce the former extreme - only one version of a library is allowed at a time. Unfortunately, this isn't possible for the OSS community where there's no enforceability.

With Duo you can actually have multiple versions of the same Javascript dependency. And then you can use the CLI to figure out what duplicates you have if you want to slim them down.

Please forgive my ignorance, but what is the problem with bundling n different versions of a library? Is it just that the bundle size increases? Or are there additional problems?

(That's not to say that the bundle size isn't important -- just that I would like to know if it's the only drawback)

Foo calls Bar and Baz, Bar calls Qux:0.01 and Baz calls Qux:0.05. Your project is Rumba, which calls methods from Bar and Baz and also needs Qux:0.2, although it turns out that you can use anything from Qux:0.03 through 0.3 and you just specified it as Qux, unversioned.

Your coworker is having problems with Baz. How many collisions are in your brain right now when you think about Qux.GiveAnother ?

Depends on the language.

Reply on news.ycombinator.com