Hacker News Reader

Comment by capableweb | Hacker News Reader

Unsure if Property Testing would actually have found this issue, 2 billion bytes is a very large string and usually you wouldn't put the limit so high, the tests would run for too long.

Fuzzing would be a more appropriate measure to have found this issue, as usually you do fuzzing with no limit of length/size, as that's the entire point of fuzzing to expand the scope of what you're testing.

Regarding test coverage, I'd argue that even caring about test coverage is giving people a false sense of security. It's easy to achieve 100% test coverage, but that doesn't actually say that you're testing the right/wrong thing, just that at one point that line of code was executed, but again, that says nothing about correctness.

View on HN

> Fuzzing would be a more appropriate measure to have found this issue

SQLite has some fairly extensive fuzzing. https://www.sqlite.org/testing.html#fuzz_testing

One of the sections on Fuzz testing:

> 4.3 Boundary Value Tests

> SQLite defines certain limits on its operation, such as the maximum number of columns in a table, the maximum length of an SQL statement, or the maximum value of an integer. The TCL and TH3 test suites both contains numerous tests that push SQLite right to the edge of its defined limits and verify that it performs correctly for all allowed values. Additional tests go beyond the defined limits and verify that SQLite correctly returns errors. The source code contains testcase macros to verify that both sides of each boundary have been tested.

It sounds from the article that "-fstack-protector-all" might remove everything except the denial of service.

"...arbitrary code execution is confirmed when the library is compiled without stack canaries, but unconfirmed when stack canaries are present, and denial-of-service is confirmed in all cases."

Everything should be compiled with all available compiler defenses:

https://manpages.debian.org/testing/devscripts/hardening-che...

It does appear safe with the OS package on CentOS 7.

  $ hardening-check /usr/lib64/libsqlite3.so.0.8.6
  /usr/lib64/libsqlite3.so.0.8.6:
   Position Independent Executable: no, regular shared library (ignored)
   Stack protected: yes
   Fortify Source functions: yes (some protected functions found)
   Read-only relocations: yes
   Immediate binding: no, not found!

It is absolutely not easy to achieve the kind of coverage that sqlite has. You need massive amounts of tests for that. Of course coverage is not sufficient, you can exercise code without checking any properties at all. But to say that it gives a false sense of security is too harsh.

> It is absolutely not easy to achieve the kind of coverage that sqlite has.

It is though. Give me one project and I'll send you a test suite that "tests" 100% of executable lines. It won't have any actual assertions, but that's not what code coverage is about anyways, so hopefully you won't mind.

Number of assertions per line of code is about as good metric for how well tested something is as code coverage, which means just about nil.

> Give me one project and I'll send you a test suite that "tests" 100% of executable lines.

I don’t think you can do that with every project. For example, a function f may do:

  if (p == null) {
    abort();
  }

If all callers also do that test, there’s no way to hit that abort() line in the program. Your test suite may not be able to hit it either. Maybe f is a private function, or that test occurs on one of its intermediate values, or f accidentally has two identical tests:

  if (p == null) {
    abort(“p is null”);
  }
  if (p == null) { // copy-paste bug
    abort(“q is null”);
  }
  if (r == null) {
    abort(“r is null”);
  }

Even if it is possible to hit that line, finding a path through the code that passes in a null value can be extremely hard.

As a simple contrived counterexample, try getting 100% coverage on a function that does if(secure_hash(foo) == 0) bar();

> Your test suite may not be able to hit it either.

Minor nitpick, process terminations are unit-testable in general; see e.g. https://github.com/google/googletest/blob/main/docs/advanced...

They don't count assertions in testing coverage: https://www.sqlite.org/testing.html#coverage_testing_of_defe...

What is fair in testing and what is being tested? If I can stub `secure_hash` and `bar`, that’s a pretty easy test to write. Assuming the symbols are visible at test time, stubbing those is relatively straight forward in C, Java, Bash, and Ruby, JavaScript, etc.

Even in the above examples, a linter should point out that the second `p == null` is inaccessible. I’m very pro test coverage, but static analysis shouldn’t be thrown out just because unit tests are available.

How much do you charge? Because adding assertions to tests that give MC/DC coverage is a lot cheaper than writing them. I know a couple of big companies who might be interested.

Compliance with the DO-178B standard was not easy for the SQLite maintainers. It required several years to achieve.

By adhering to the DO-178B standard, the testing is confirmed extensive and satisfies requirements for use in avionics.

"Rockwell Collins... introduced me to this concept of DO-178B. It’s a quality standard for safety-critical aviation products, and unlike so many quality standards, this one’s actually readable... one of the key things that they push is, they want 100% MCDC test coverage.

"Your tests have to cause each branch operation in the resulting binary code to be taken and to fall through at least once.

"At the machine code level... actually, MCDC’s a little stricter than that. Let’s not dwell on the details, but I had this idea, I’m going to write tests to bring SQLite up to the quality of 100% MCDC, and that took a year of 60 hour weeks. That was hard, hard work. I was putting in 12 hour days every single day. I was just getting so tired of this because with this sort of thing, it’s the old joke of, you get 95% of the functionality with the first 95% of your budget, and the last 5% on the second 95% of your budget. It’s kind of the same thing. It’s pretty easy to get up to 90 or 95% test coverage. Getting that last 5% is really, really hard and it took about a year for me to get there, but once we got to that point, we stopped getting bug reports from Android."

https://corecursive.com/066-sqlite-with-richard-hipp/

Indeed, SQLite is used for flight systems on the Airbus A350. No other major database vendor is certified to do so.

https://www.sqlite.org/famous.html

> It's easy to achieve 100% test coverage

If code hasn’t been written to be testable, 100% coverage is not _easy_, by any stretch. Any legacy code base written without unit tests that I’ve come across required significant refactoring to get any basic assertions I place, let alone any reasonable coverage.

I also think there is some value in high coverage, even without assertions. At a minimum it’s a guarantee that code has been run before production/distribution. That’s probably a lot more than can be said about a lot of code.

Yeah, exactly. People tend to misunderstand fuzzing, and think of it as "feeding random garbage at an API". Fuzzing, done with proper tooling, is automated coverage testing. You have a bounds overflow that depends on input data but only if condition A and B and C are true in the runtime state? Fuzzing will find it, almost always, by detecting the branching behavior of your code.

So all the replies here saying "the tests didn't find it because they didn't cover the case" are missing the point. There are test paradigms that don't rely on humans to enumerate cases.

> Unsure if Property Testing would actually have found this issue, 2 billion bytes is a very large string and usually you wouldn't put the limit so high, the tests would run for too long.

Yet, this is a textbook case of issues caught by static analysis.

Coverage testing is just a special case of a more general concept: mutation testing (where the mutation is "if you reach this line/branch, blow up.")

Perhaps mutation testing could have revealed the test suite was lacking in a test detecting that bug (for example, a mutation that changed a type from unsigned to signed or vice versa.)

I'm so sad that so few test frameworks/libraries offer mutation testing. (And its profiling cousin Causal Profiling.)

Admittedly the some languages might make them very hard to do, but the payoff would be so, so sweet.

My team at work took it out of our product. :( I couldn't make strong arguments that it was providing value, as it had turned into the slow test that always passed, by the time new folks joined. I do think it has a lot of merit, though.

To be clear... mutation testing is a way to assess the power of a test suite, not a way to test software itself (in this it is analogous to using statement/branch coverage to assess a test suite). Do you mean the tests added to achieve high mutation coverage were not worth the time it took to run them? The mutation process itself is not something that would be part of CI. Doing all that mutation and generating the tests to kill non-equivalent mutants is very time consuming, I will admit.

On the code base in question, the full but with mutation tests was only 8 minutes. Not snappy, but hardly a disaster.

I'm guessing these tests were run constantly, in CI, not periodically (say, daily, or even weekly, or before release). SQLite has tests that take many hours that must pass before release.

There are testing modes that are worth doing, and I think should be part of a development process, but shouldn't be run at every check-in. Long soaks of random input property-based testing is like this, for example.

We run a lot of tests constantly. So, yeah. That said, as soon as it is in the release pipeline, I really encourage the team to not directly watch it. If we don't trust our tests to allow a change to go out, we need to fix the tests. Not train members to watch the deployment.

It's not a matter of watching a deployment, it's a matter of having separate testing modes used in appropriate ways. Rapid tests should catch most bugs in CI, but more laborious tests could catch other bugs that would escape CI.

Personally, there's a free software effort where my contribution is running tens of millions of randomized property-based tests each day and registering bugs when failures occur. This supplements and complements the fast tests that all must pass before changes are merged to master.

Apologies, I meant my point to be tangential. I agree with your general point. Our deployment pipelines go from build to prod though. With a general cadence on my current team of less than an hour. 6ish minutes to that is not a concern.

Even when I was on teams where the general cadence of getting changes out all the way to prod was on the order of a week, we still had changes getting to our beta environments in less than an hour. And that was on a team that wanted mutation testing, so we had it.

Edit: So my point was team mates that were worried about the 6 minutes added by mutation testing there, were the ones that would commit and then constantly refresh the page so that they could "approve" a change as it goes out. I pushed for all approvals to be automated, so there was nothing for them to watch.

Boundary Value tests are standard practice when doing extensive testing.

Bugs rarely occur for typical input values but more often occur at the "boundaries" or limits of the input domain.

That being said I don't really fault SQlites testing here.

The key takeaway for me is that even with exhaustive testing you can still have security issues like this, so perhaps what is needed is a language or methodology change.

If SQLite, a quality codebase with exhaustive tests, can have memory vulnerabilities, maybe we need to leave C behind.

SQLite is used in places that Rust has no compiler for and that no other language has a compiler for.

One severe CVE in about 20 years is no reason to rewrite in another language and remove support for some platforms.

In the long term, I hope C stops being the default language that hardware vendors provide for their hardware.

Me too, but I also hope Rust is not that default language.

Just how many machines are we talking about? Does anyone have estimates?

Probably every Airbus A350 XWB and a lot of GM, Nissan, and Suzuki cars. Maybe also a few or a lot of General Electric products. [1]

The Library of Congress also recommends it for preservation of digital content, which would be harder if Rust compilers were lost. If all C89 compilers were lost, it would be much more simple to create a new one to compile SQLite than it would be to recreate a Rust compiler.

[1]: https://www.sqlite.org/famous.html

I'm not saying we need to rewrite SQLite and I'm not sure why everyone is assuming I suggested rust.

But people need to consider how much testing SQLite did and still hit memory vulnerabilities when they consider picking up C for a new project.

Your problem is that you need to replicate the DO-178B certification in Rust:

"I’m going to write tests to bring SQLite up to the quality of 100% MCDC, and that took a year of 60 hour weeks. That was hard, hard work."

Whatever comes out of Rust will be much worse until compliance is achieved. That's going to take substantial time.

Reply on news.ycombinator.com