Hacker News Reader

Comment by dignati | Hacker News Reader

> What I've noticed in practice however, is that occasionally, this process will allow an upgrade to a dependency that will pass the automated build and test step, but introduce the wildest runtime error into the application. Usually at the time when we aim to deliver something.

Sounds like dependabot is very useful for uncovering insufficient test coverage or missing integration tests :)

View on HN

That would be a shallow reading, however. The last two major runtime issues wer actually one that broke the test runner and ignored a number of tests. And another runtime error was a Python Django specific sub dependency that broke the admin interface, which obviously, we don't explicitly test.

On the other hand, very recently, we had to abort a release, because of an outdated dependency that Dependabot DID actually raise.

Which is why I don't want to throw the baby out with the bathwater, as one or two people have suggested.

But I can say that I think that the reality of working with Dependabot, is not very well reflected in popular online articles.

> the admin interface, which obviously, we don't explicitly test

This... is not obvious. It broke a part of your software which you care about. You care, it caused a problem, so it should be tested.

You routinely write unit tests for dependent packages in your apps?

This is not testing some other package. It's testing functionality provided by your app that happens to rely upon a 3rd party package. The 3rd party package has its tests, but doesn't know or care about the specific integration environment of your site.

With regards to writing tests of the Django admin, you need to do it if your site is customizing and/or depending upon it.

No. This is what integration tests are for.

These kinds of failures are super annoying, but it’s the sort of thing that e2e and canary releases should help you get some coverage on.

Despite the marketing, unit tests are pretty fragile, especially in code with lots of deps (another good reason to limit deps)

Yes, we routinely write tests that interact with dependencies to make sure our application works.

But once you've imported that package it's part of _your_ codebase.

While this is true, many will probably disagree, just because they don't want to consider the maintenance burden that external dependencies will introduce.

So between choosing to write everything themselves (and getting nothing done), writing tests against dependencies (and getting little done due to the overhead), or claiming that external dependencies should have tests of their own, many will pick the latter.

Then again, in a world where create-react-app results in 180 MB of dependencies and about 1500 modules (probably different numbers now, using some older ones from my blog post), auditing security is an uphill battle, not even talking about actual testing.

The situation in the back end development, isn't that much better either, to be honest, because once you look into the complexity of any framework like Spring, Laravel, Django, Rails etc., it becomes apparent that creating a fully featured framework like that is a huge undertaking.

That said, you should at least test the bits where the external dependency is integrated with your codebase.

> one that broke the test runner, and ignored a number of tests

That's unfortunate! For the project I'm working on, we've "solved" that by showing the number of test and the difference to the number of tests that ran on main.

FWIW, at previous jobs, upgrading Java dependencies was a major pain because they were all outdated and the latest versions introduced too many breaking changes for us. At my current job, we pretty much instantly merge all PRs from dependabot because we trust our CI. Upgrades rarely introduce problems and if they do, they are easy to fix.

> Python Django specific sub dependency, that broke the admin interface, which obviously, we don't explicitly test.

There's your problem.

Was it an update to the test runner or test specific packages that broke the test runner? I would ignore infrastructure/testability/tooling packages in dependabot and do them manually to prevent these errors.

Most test runners have an option (or can be easily modified) to fail when 0 or less than X tests have been run. You should use it for situations like this.

But what if there's also a bug where that feature doesn't work :)

What if there's a bug where your CI ignores all steps? There's always some situation possible and we can go all the way to nasal demons. You have to accept the risk at some point.

Reply on news.ycombinator.com