How is LLVM tested?
lowlevelbits.org
lowlevelbits.org
Well, no, and i'm not sure where they got this. It's not a "decision". BTW, This also assumes that all of the tests in the unit test dir are unit tests, and all the tests in test/ are regression tests.
This is explicitly false.
The general viewpoint of LLVM is that most things people write (analysis, optimization) should be tested by checking the IR or dumps. That is, we want easy to follow and modify unit tests, not just compilable C++ code ones.
Thus, you will find a lot of unit tests that are in test/, instead of unittest/, because they are testable using IR printers.
Those things that are API's should be tested by unittests, and work is underway to increase coverage. The reason some things aren't cover is simply that in the earlier days of the project, not all of this got unit tests, and it's only in the past couple years that people have become sticklers about it.
So that one would be knowingly "not enforcing good development practices", not "llvm internals change all the time". The fact that internals change all the time is in fact a good reason to have unit tests.
Right, and it's infinitely easier to just dump the result of the analysis and check that it said what you want using lit.
We even have a framework explicitly to do that -analyze for opt is specifically meant to run analysis and dump their output.
This doesn't test API boundaries, but the dumpers and printers are usually forced to be written in terms of the API.
(and folks are starting to get religion about testing the other API boundaries)
SQLite ( https://sqlite.org/testing.html )
Opus ( https://www.ietf.org/proceedings/82/slides/codec-4.pdf )
I don't mean it as criticism of the LLVM project, as I haven't worked with it and can't say whether its tests are sufficient. Only that, from reading this overview, its testing techniques do not seem particularly elaborate.
Opus testing is impressive:
> Run thousands of hours of audio through the codec with many settings
- Can run the codec 6400x real time
- 7 days of computation is 122 years of audio
So, yes, the testing of LLVM is &quite& elaborate.
Oh, sqlite uses this fuzzer to test itself.
"Amount of unit tests is pretty small comparing to regression tests. One reason behind that decision is that LLVM internals constantly change all the time. Supporting tests under such conditions is very time consuming."
Isn't the whole promise of testing is easier maintenance in the long run? Also, regression tests seem to run much faster, is it because of the Google Test?
Example: I was extending some code the other day that had 3kloc of thin wrapper code and 30kloc of mocks and unit tests which effectively did nothing but ensure that the 3kloc passed things through unaltered. Meanwhile, the code is widely acknowledged to be frustrating to interface with because it uses a string-based API that isn't documented (and no, those unit tests won't tell you semantics), versioned, or maintained with an eye towards compatibility. Naturally, there are no integration tests with the layer above or below it, and those interfaces are where all the actual bugs happen. So the tests increased cost by a factor of 1000% and had 0 return.
Also see: cargo cult comments. You know, these guys:
/**
* Gets the foo.
*/
Foo getFoo() {return foo;}
Fixing the problem of unit tests becoming de-facto integration tests is exactly what mocks are for. If you don't mock your dependencies, then you are in fact executing an integration test. The problem becomes that your mocked dependency and the real dependency can now drift because there's nothing tying them together. So a unit test works but you still have an integration bug.
Personally, I think both unit and integration tests are really important, but that integration tests tend to get a bit underlooked with all the existent zeal towards unit tests. Testing a component's contract is obviously necessary, but testing that your components are expecting the correct contract from others is also important.
As always, exceptions: algorithms, logic hairballs, parsers, anything with a sufficiently low surface-area:volume is likely to generate significant bugs in the interior, so it can likely benefit from a unit test.
Well, it's not as if the point is regarding testing vs not, it's regarding the balance of what kind of tests LLVM should have.
LLVM and Clang have extraordinarily well-defined interfaces at their boundary (C/C++/LLVM IR|executables), so writing regression tests is bound to reap great value over time. Writing unit tests, less so. The benefit of unit tests over ones with larger scope is that it's often easier (or possible at all) to write tests that cover uncommon use cases/environments/errors.
A less tangible benefit of unit tests is that it helps guide sufficient decomposition of the design-under-test. From what I've seen, LLVM doesn't need any help in this area, they're extraordinarily well-disciplined here.
It depends on the kind of maintenance. It makes it easier to add or refactor code without changing the semantics of what exists, but it makes it harder to modify the semantics.
A test suite that's too tight can be an impediment to change.
Yes, but the structure of the tests matters. Generally speaking there are two major problem areas to avoid with unit tests:
1. Unit tests that are too close in design to the specific implementation. What this means is that instead of testing the desired behaviour in an abstract way, you mirror the design decisions made in the code being tested. When the code being tested changes, this can break the unit test, even if the new code has the desired behaviour.
2. Test-induced design damage. Unit testing relies on being able to isolate the code being tested. This can lead to an increase in code complexity, as well as a less flexible design.
Both of these issues can be overcome, but it does mean that you have to consider what to test and how to test it, as badly applied tests can cause maintenance issues.
In contrast, the regression tests don't have the dependency on implementation detail, just the interface.