Comment by btilly | Hacker News Reader

Comment by btilly | Hacker News Reader

Parent Full thread

Here is my beef with Debian.

A lot of the software they package comes with unit tests. Those unit tests have a purpose. They are meant to see whether or not the software as configured and installed, works.

Debian systematically strips those unit tests out, and never runs them to see how much stuff they are shipping already broken. Why? Why not package the unit tests as an optional package, and make sure they have a wide variety of systems in different configurations running them to notice problems?

I can't count how many times I've tried to install a Perl module from CPAN, found it was failing its unit tests, installed the Debian package with it, ran the unit tests, and found that the unit tests said the package, as installed, was broken. It's not as if the package developer didn't try to tell you you were missing something. Why throw that information away?

If they did this then they'd automatically catch a lot of problems that currently get missed. Heck, insert a test for ruby gems that says, "Does this software start?" They'd have caught this bug automatically.

Until Debian catches up with standard best practices, I completely can't agree with the meme that they run software better than everyone else. It isn't as if unit testing is a new idea. It has been pretty mainstream for most of the last decade. Some were doing it earlier. Perl has been doing it since the 1980s.

I used to maintain Debian packages. As far as I know (or could find looking through the Policy manual and a few language-specific packaging guidelines), there is no policy against running tests after builds. I spent a few minutes looking through several packages and didn't see any that ran tests, but I have seen bug reports about test failures. If there are tests and they run using a reasonable amount of resources, I can't imagine anyone would argue against running them. The package maintainer has to do it, and I assume that some mess up.

I don't think it's so cut and dry. In the ideal case, the packages don't need tests because they are going into the system already tested. If the package is correct then running the unit tests is just redundant work because you are getting an identical copy of a package that already tested good with the appropriate dependencies. I think that's the theory anyway. On the other hand, theory does not always match reality.

Debian CPAN packages do run the unit tests--they just run them when you create the package and not when you install the package. In fact, the package will not build unless the tests pass.

Sorry, but it really is that cut and dry. In the ideal case you don't need unit tests. In the real world, they are useful.

One of the big things that unit tests are supposed to catch are hidden dependencies. By nature, hidden dependencies are likely to be installed on the maintainer's machine, so unit tests only help if they are run somewhere else.

As a prime example, the bug that Zed was complaining about was a dependency problem of EXACTLY this sort. It worked fine on the maintainer's machine. It didn't work on Zed's machine because he didn't have a key library installed. No unit test the maintainer can run would catch this. Even a stupid unit test Zed can run would have flagged this immediately.

Incidentally about the Debian CPAN packages, my experience with them a few years back was that they were so consistently broken (usually because of locale issues) that I have a hard time believing that unit tests had actually been run by anyone. Or if they were, they were ignored.

Well, then maybe what they really need is a single server/farm out there that just goes through all the perl/ruby/whatever modules and installs them and runs their unit tests. That way you get the same coverage but you don't waste my time. I, personally, don't want a bunch of unit tests to run when I install stuff--it's slow enough as it is and it's (usually) completely redundant.

It seems the same to me as slackware or one of the other "compile everything yourself" distros--they force you (the installer) to do a bunch of redundant calculations. If someone can compile on my architecture then why should I ever have to compile it? The same with unit tests--if someone can run the unit tests on my exact configuration then why should I ever have to run them?

You still don't get it.

If you're installing item after item, then the odds are that at some point you're going to install the dependencies, and after that the tests will pass and mean nothing. So you keep on wiping, installing one thing, and testing. Fine. Now you have cases where package A conflicts with package B and you never find it. And now what? The number of combinations to test is incredibly large, and nobody is likely to have tested your exact system.

As for the unit tests, make them available, make them optional. That's OK by me. I'd opt to use them. Particularly since I learned the hard way that most of what I want in CPAN either isn't in Debian, or isn't up to date there. So I install things directly from CPAN. That means that when I install from Debian, there is a possibility of library conflict that I want to notice.

Perhaps a compromise would be to have the package-install-time unit tests run in "testing" and "unstable", but be turned off for "stable"?

That would be worlds better than what we now have. But I'd like the option to turn it on. (And I'm sure others want the option to turn it off.)

I'm not sure what the -dbg packages in Debian/Ubuntu are strictly for, but wouldn't this be a case for it? If one could post-run the tests by saying "apttest Foo" or something along those things, integration into utilities to automatically install test packages and run them shouldn't be a huge hurdle.

I can't speak as to whether or not it would be a good idea for -dbg packages to run unit tests, but their nominal purpose is that they contain extra symbols to allow for debugging.

http://wiki.debian.org/DebugPackage

> A lot of the software they package comes with unit tests.

I don't know about 'a lot', but in any case, your idea is potentially a good one, although probably not 'low hanging fruit' in terms of the amount of work it would take to get all the various systems integrated and talking together.

And in terms of 'beef with Debian', it seems to be a general beef with modern OS's, because none of the others do it either, do they?

Others don't tend to share unit tests with end users. But I'm quite sure many proprietary OS vendors run regression tests on a variety of variations on their platform. So while not perfect, on this particular item I think they do better than Debian does.

> So while not perfect, on this particular item I think they do better than Debian does.

Could well be, but at this point, we're moving pretty far away from Zed's spittle-flecked exhortation to pursue "in-person confrontations meant to embarrass package maintainers, SEO tricks to get people off Debian, promotion of any alternative to Debian, anything to make them pay" and merely talking about what could be made better (and there's always something).

You don't think that it is embarrassing that package maintainers are given a set of unit tests that enable them to tell that things are correctly packaged, and they throw it away?

I'd be embarrassed by that!

Fedora has a few *-tests packages, e.g. ipython-tests which has the following description:

    This package contains the tests of ipython. 
    You can check this way, you can test, if ipython works on
    your platform.

Reply on news.ycombinator.com