Hacker News Reader

Comment by mastazi | Hacker News Reader

Usually, dependency hell doesn't bite you, until it does. Try to rebuild that thousand-dependencies app in three years from now and you'll see ;-)

I recently had to rebuild a large RoR app from circa 2011 and it took me longer to solve dependencies issues than to familiarise myself with the code base.

Excessive dependencies are a huge anti-pattern and, in our respective developers communities, we should try to circulate the idea that, while it's silly to reinvent the wheel, it's even worse to add unnecessary dependencies.

View on HN

> Try to rebuild that thousand-dependencies app in three years from now and you'll see ;-)

Let's be honest though, in the current trendy javascript ecosystem these people will already be two or three jobs away before the consequences of their decisions become obvious. Most of the stuff built with this is basically disposable.

Not to mention a good chunk of the companies they were writing this type of code for won't exist by then either...

I never can believe how often frontend developers talk about "you're just going to rebuild it all in 2 years" anyway. I guess it's a good way to keep yourself employed.

Shouldnt the Gemfile.lock have prevented that? It takes your gemfile and makes a specific-version locked file of those gems.

The gemfile.lock must have been "gitignored" at some point, because it had much older packages than the ones in Gemfile. Background: all we had was a git repo and did not have access to any "living" installation.

> Try to rebuild that thousand-dependencies app in three years from now and you'll see ;-)

This is your fault for expecting free resources to remain free forever. If you care about build reproduction, dedicate resources to maintain a mirror for your dependencies. These are trivial to setup for any module system worth mentioning (and trivial to write if your module system is so new or esoteric that one wasn't already written for you). If you don't want to do this, you have no place to complain when your free resource disappears in the future.

Most of the time this is something you inherit. Not something you wrote.

Yes that was my case.

A maintenance programmer should be raising to management the risk if they do not have reproducible builds.

The issue isn't that the company's software has a dependency. The issue is that the company is taking for granted the generosity of others. If they did not get a reproducible build before, they should attempt to get one as soon as they are aware of the problem. If the package is no longer available, they must now accept the punishment in terms of lost staff time or dollars to work around the lack of the dependency.

I agree with you, but s..t happens.

I agree. But I find two problems with your proposal:

1- Maintaining a mirror of dependencies can be a non-trivial overhead. In this app that I was working on, the previous devs had forked some gems on github, and then added that specific github repo to the requirements. But they did not do it for every dependency, probably they did not have time/resources to do that.

2- As a corollary to the above, sometimes the problem is not the package itself but compatibility among packages. E.g. package A requires version <=2.5 of package B, but package C requires version >= 2.8 of package B. Now I hear you asking "then how did it compile in the frist place?" probably the requirement was for package A v.2.9 and package C latest version, so while A was frozen, C got updated. This kind of problems is not solved by forking on Github, unless you mantain a different fork of each library for each of your project, but that's even more problematic than maintaining dependencies themselves.

P.S. At least for once, it wasn't "my fault", I didn't build that app LOL ;-)

> 1- Maintaining a mirror of dependencies can be a non-trivial overhead. In this app that I was working on, the previous devs had forked some gems on github, and then added that specific github repo to the requirements. But they did not do it for every dependency, probably they did not have time/resources to do that.

You've precisely identified the trade-off. You basically have three options. You can

1. Maintain a local repo of your dependencies (high effort)

2. No dependencies, include everything as 'first-class' code (lower upfront effort, but v. messy)

3. Rely on third-party repos (easiest, riskiest)

Yes I agree, yours is a very good summary of the main alternatives at play. They all have pros and cons as you have rightly noted.

This problem is solved by mirror dependencies and pinning the versions. Even against a git repo, pinning to a particular sha is something that is possible.

Automatically upgrading versions (i.e. not pinning versions) in a production build is an anti-pattern.

These sound like problems incurred due to a previous lack of software engineering rigor. As an industry, when we encounter challenges like this we should be learning how to solve them without reinventing the wheel. Pinning versions and maintaining mirrors of dependencies (whether that's an http caching proxy for npm/pypi/maven/etc or keeping a snapshot of all dependencies in a directory or a filesystem somewhere) is something that any company requiring stability needs to take seriously.

Of course pinning the versions and identifying the particular commit in the Gemfile would have solved it, as long as it was done for every package, otherwise we are back at problem n. 2 in my post above.

In this particular case, there were just 3-4 requirements (out of more than 100) that were pointing to a git repo, and only one of them also specified a particular commit. The other "git-requirements" were just cloning the latest commit from the respective repo.

> Automatically upgrading versions (i.e. not pinning versions) in a production build is an anti-pattern.

We did not have access to a production version, only to a git repo, that's the very reason why we had to rebuild in the first place. I can imagine all versions were locked when the system went into production years ago.

So, in the context of this discussion... you should make use of micro-modules to reduce code duplication, avoid defects, etc. However, don't expect those micro-modules to be maintained or available in the future; so you need to set up your own package cache to be maintained in perpetuity.

Or, you can implement the functionality yourself (or copy/paste if the license allows) and avoid the hassle.

Yeah, I think the takeaway here is something like: "find the right middle ground between reinventing the wheel and dependency hell"

There's more to dependency hell than "oops, the package disappeared." Try updating one of those dependencies because of a security fix, and finding that it now depends on Gizmo 7.0 when one of your other dependencies requires Gizmo < 6.0.

Yes that's exactly the kind of issues that I had experienced.

I've been in the same situation as OP many times (although in most cases I've been brought in fix someone else's code).

In the Ruby ecosystem, library authors didn't really start caring about semantic versioning and backwards compatability until a few years ago. Even finding a changelog circa 2011 was a godsend.

I think this was mainly caused by the language itself not caring about those either. 10 years ago upgrading between patch releases of Ruby (MRI) was likely to break something.

At least this is one thing JavaScript seems to do better.

Why did you have to rebuild it?

Long story... we had the git repo but lost access to any working installation. I had to rebuild a dev vagrant VM first, and later on a production-ish setup on a server.

PS I did not downvote you and I find your question legitimate...

I can't speak for him, but upgrading really old Rails apps can get complicated very quickly. Especially when you're going across multiple major versions and have to deal with significant changes in Rails behavior and broken gems. "Rebuild" might not be the most accurate way to describe the slow, steady incremental approach you're forced to take (you aren't redoing huge swaths of your domain logic, for instance), but it gets the gist across.

The same goes for almost any legacy app.

Rebuilding for most cases seems silly to me.

I've been working on a .net web app, that's been around since 2008. It's been continually evolved, so it's running on the latest MVC framework, uses microservices etc

As result it's build up a huge amount of automated tests. The business logic has been built up from experience and is well tested even for odd cases.

You throw away a lot of stuff for a rebuild.

Oh, definitely. It's not something I'd recommend ever doing for the hell of it unless you're a real masochist. Even then, it might be easier if just broke out the floggers.

All kidding aside, if the concern is just for continued security patches, there's always Rails LTS [0]. You have to pay to play, but it's cheaper than a security breach or weeks of development time. But if you're dealing with significant performance issues or looking at major feature changes/additions, it might be more effective in the long-term to consider an upgrade. You just need to be aware of just how large a project that can turn out to be.

0. https://railslts.com/

I could not not-rebuild... All I had was a git repo.

I find large dependencies like RoR itself cause a lot more dependency hell than zillions of small dependencies like this one. What kind of dependency hell could possibly happen for a module like this?

Reply on news.ycombinator.com