Hacker News Reader

Comment by chowells | Hacker News Reader

Would your aarch64 test suite have caught it? Testing is fine and all, but it can only find bugs the test authors predicted. There will always be some issues that test suites miss.

It's obvious to add regression tests for this and related issues now that they've been found. And great! That's what regression tests are for. But testing is always going to be fundamentally incomplete in that it can only cover non-obvious things after they've already been encountered.

View on HN

I would've expected a compiler to have a huge integration/regression test suite which just consists of other pieces of software's test suites. Grab the N most popular Haskell programs with test suites, compile them, run their test suites, verify that they still pass on the new version.

Isn’t rust doing something like that? Can’t seem to find the article right now but I remember reading something about this approach.

Yes, “crater” is the name of the tool.

> Crater is a tool for compiling and running tests for every crate on crates.io (and a few on GitHub). It is mainly used for checking the extent of breakage when implementing potentially breaking changes and ensuring lack of breakage by running beta vs stable compiler versions.

https://rustc-dev-guide.rust-lang.org/tests/crater.html

https://github.com/rust-lang/crater

https://crater.rust-lang.org/

Even then such a bug as GHC's can conceivably not be detected.

It would be much harder to achieve this with GHC since the package manager is far from deterministic.

I'm not sure what you mean by this. `cabal-install` is quite deterministic; you can even freeze the index date for full build plan determinism.

The largest challenge in this area is the fact that core libraries' interface sometimes change. Our [head.hackage](https://gitlab.haskell.org/ghc/head.hackage/) infrastructure is one way of addressing this. I agree that it would be great to leverage `head.hackage` to test GHC against a larger body of packages.

It definitely _can_ be deterministic, but my experience of pulling down packages without a frozen index is that it rarely compiles. This, inevitably is because the plan depends on the current state of hackage. A combination of “it might pick a different minor version”, “different versions can cause cascading effects”, “many libraries have breaking changes” and “Haskell is extremely unforgiving of potential breaks” makes it very difficult to use without either a) being really good or b) downloading a freeze index. (I download freeze indexes.)

The whole core libraries thing is difficult, in that I can see it’s a real problem for everyone constantly updating their packages (and consuming them), but also that most of the changes are actually good ideas.

nixpkgs

The issue is that basic math doesn't work. I don't dispute that no test suite can catch everything, but yes I do expect that any reasonable test suite that compiles and runs code would have caught this.

Basic math that does multiple operations on the same 8 or 16 bit value.

I agree.

I'm still confused, but I hope this is correct. IIUC the problem here is that they do something like

  (a // b ) * b

where // is the integer division. They are not breaking all math operations, just math operations that have a very specific pattern.

Also, I'm not sure if it's important that 128 <= b <= 255, because the main problem seams to be they are extending a number and the leading 1 in the binary representation is a problem (???).

So you must test a very specific math pattern, with very specific numbers, and then hope that other parts of the optimizer don't optimize the problem away before the bug in the compiler makes a mistake.

I don't know too much about the automatic test suit in Haskell, but I know more about the automatic test suit in Racket. In Racket there are a lot of test written manually, and some randomized test, and also tests for many of the packages. But I really doubt there is a test somewhere for this specific case.

[Is there a cross language regression test collection? Something that collects weird cases, so other languages maintainers can take a look, translate the code and add it to the tests.]

That is correct. The problem is only triggered by very particular patterns of integer operations which happen to get lowered to C-- where an operand is bound to a local register; this register then gets inappropriately mutated by the code generated by the backend.

As you say, GHC, like most language implementations, has an extensive [testsuite](https://gitlab.haskell.org/ghc/ghc/-/tree/master/testsuite) (10k tests which result in over 30k testcases). Sadly, none of these tests covered this particular case.

Moreover, we also have a systematic [property testing framework](https://gitlab.haskell.org/ghc/test-primops) testing the backends and primitive operations provided by the compiler. This testsuite also didn't test this form of program since it focuses on testing the expression fragment of the Cmm language (and therefore won't generate local register bindings). This is a blind spot that I am going to look into addressing.

Reply on news.ycombinator.com