Comment by runeks | Hacker News Reader

Comment by runeks | Hacker News Reader

Parent Full thread

The issue I have with tests, is that I feel like I'm just implementing my algorithm a second time, in a different wording (language/domain), hoping that this will catch some unknown bug(s).

If a test suite covers all aspects of an implementation, it's basically a second implementation, and then the question becomes whether the time spent writing this test code would actually have been better spent looking over the actual code that defines your algorithm. If we can't detect errors in code by reading it, I don't think tests will help us much.

Furthermore: if our algorithms require a second "test implementation", shouldn't we also write a test for the test, to confirm that the written test actually tests correctly? What good is a broken test? If you've built a plywood door, and a machine for testing that door, wouldn't it be useful to have a machine that tests that your testing machine actually tests the door properly?

All jokes aside, I really enjoy automated, high-level tests, which mimic actual user behaviour. I think this is different because we're no longer implementing an algorithm a second time around (as a test), but rather encoding user behaviour (of the algorithm) into an algorithm (new code).

> If a test suite covers all aspects of an implementation, it's basically a second implementation, and then the question becomes whether the time spent writing this test code would actually have been better spent looking over the actual code that defines your algorithm. If we can't detect errors in code by reading it, I don't think tests will help us much.

The OpenBSD code has been audited many times by very experienced and careful C experts. Fuzzing apparently found the very few things they hadn't already found by reading the code.

And it if I understand OpenBSD goal correctly, they take any kind of edge they can get to find those missed parts.

Their formula for what is worth essentially boils down to "if this approach can find at least one bug no other approach would do it, we will use it".

> The OpenBSD code has been audited many times by very experienced and careful C experts. Fuzzing apparently found the very few things they hadn't already found by reading the code.

That's definitely comforting, but I guess my point is that the fact that bugs can even hide in plain sight in the first place is the root cause of the problem. If we can look at a specification of what something is supposed to do - the code - and not see that it does something entirely different, then the problem is with the programming/specification language.

The very purpose of a programming language is to make the specification of computer programs readable (rather than reading assembly). If a specification language needs a a computer program to go through the specification and see if it specifies things that we didn't intend to, then that specification language misses the very point of being a language in which to specify things in the first place.

I don't think it's that simple. A lot of bugs are the result of disparate pieces of code -- each of which is fairly easy to verify by reading it -- interacting in an unforeseen way.

I would argue that this effect is not a product of code but of global state. If you have a pure function that transforms some input to an output, all you can do wrong is use the wrong tool for the wrong job, which the type system should, preferably, prevent you from doing. Using the wrong function for the wrong job means we haven't restricted well enough the input the function takes, through clearly defining the types of the arguments, not allowing for meaningless invariants.

If a specification language requires the writer/reader to be aware of the entire specification in order to verify whether an isolated piece of specification is valid, that specification language is not of much value, I would argue.

It occurs to me that the problem with traditional languages is that they do not allow the direct reference to information (values); only to variables which contain values (information). In Haskell, if you want a place to store information you create a TVar (variable), while you are forced to do this, always, in traditional languages, if you want to work with information/values properly. You're forced to keep information in a register if you want to manipulate it, and always refer to the register, when the information it contains is what you're really concerned with. Having to both think about the values/information and where you stored this information is double work. Why not reference the information directly?

I agree that many bugs occur due to state. I disagree that languages, particularly imperative languages (which you seem to be attacking here), are significantly responsible for bugs.

Syntax is not really an issue for any programmer with a reasonable amount of experience. I very rarely have trouble accurately transcribing ideas to code. More often, my ideas are faulty.

Physics engines in games are a perfect example. They're a perfect use case for functional programming, there's not a lot of complex state ideally, and yet even simple simulations are notoriously fraught with unexpected behavior. See: thousands of YouTube videos of physics glitches in games.

    > Syntax is not really an issue for any programmer with a reasonable amount of experience.

    > They're a perfect use case for functional programming, there's not a lot of complex state ideally,
    > and yet even simple simulations are notoriously fraught with unexpected behavior. See: thousands
    > of YouTube videos of physics glitches in games.

I'm not sure I'm following. Are you arguing that because physics engines are a perfect use case for functional programming, that they are implemented as pure functions? I don't know of any physics engines that are implemented this way, hence the buggyness.

If an implementations of 3D scene rasterization contains glitches, the language used is most definitely the issue, because producing glitch-free 3D animations is a solved problem. No programmer writes a physics engine to produce glitches, so if they magically appear, even though they aren't visible in the spec, the spec language is faulty.

I guess I'm arguing that the problem is that most popular languages allow you to write programs/specs that are invalid (will crash when executed). So we're writing specifications in languages where we can't even say whether the specification is valid or not (actually implementable). If humans produce buggy programs, that is proof that we're using the wrong tool, I'm arguing, because no one intends to produce bugs.

Haskell is a bit scary, at least it was for me in the beginning, because it can seem so hard just to get your program to compile. But that's because the compiler actually verifies that your program is valid - that no unhandled exception will occur. If people are having writing valid, purely functional programs, perhaps this is, in part, because their idea is unimplementable, and Haskell is telling them this via not being able to compile your spec, while other languages inform you of this via an unhandled exception that crops up a year later, after the code is in production.

> the compiler actually verifies that your program is valid - that no unhandled exception will occur

That's not actually correct. The compiler verifies (modulo your transitive use of unsafe primitives) that certain things won't occur. Unhandled exceptions are not one of those things.

That depends on how you define an unhandled exception :)

You're right of course: no compiler can guarantee lack of exceptions when it comes to IO, but I would argue that GHC, indeed, does verify that - unless a function explicitly throws an exception - no exception will occur. This is very different from C, where an exception can be the result of just about any operation (if you do something wrong), rather than only an effect that appears when you use "throwIO" (or similar). It's the difference between exceptions being used as a tool by the user (programmer), rather than a way for the compiler to tell the user (after your program is compiled and is running) that it has no idea what to do now, and will abort. In Haskell I write the "error"/"throwIO" statement causing an abort, in a C program I might have intended something completely different, but the compiler throws one at runtime.

We mustn't mix up IO with everything else, because reality is inherently unreliable, and it's just the nature of it that a certain file won't necessarily be readable tomorrow - we can never know that. But we can know whether a pure function, sans compiler bugs, will execute without throwing an exception. And I would argue that adding "error" or "fail" does not constitute an unhandled exception, as that statement was put there intentionally by the author of the program (perhaps to test out something).

> No programmer writes a physics engine to produce glitches, so if they magically appear, even though they aren't visible in the spec, the spec language is faulty.

I'm saying that popular physics engines generally have bulletproof, tried-and-tested code, and are even implemented in a very functional way on a conceptual level. Despite this, they exhibit glitches. The glitches don't appear magically out of the spec language, they are an inevitable result of the discrete nature of realtime physics simulations.

This is meant to be a counter-example to your assertion that bugs come from miscommunication between computers and humans. My argument is that more often than not, we communicate our ideas perfectly, but our ideas are flawed. In the case of physics engines, they are flawed by design in order to compromise accuracy for performance.

    > In the case of physics engines, they are flawed by
    > design in order to compromise accuracy for performance.

This is an odd definition of "flaw" to me. I would call it a design choice, compromising accuracy over speed, which we're forced to do always, since we don't have infinite compute power. Would you call H.264 video and MP3 audio flawed by design because they prioritize bandwidth reduction over lossless reproduction?

    > The glitches don't appear magically out of the spec language,
    > they are an inevitable result of the discrete nature of
    > realtime physics simulations.

You seem to be arguing both that glitches in physics engines are inevitable, and that they're a design choice (sacrificing precision over speed).

I think the difference is that the test shows what the idea you (as the architect) have is, while the code shows how you (as the developer) implemented it.

And then you run the tests and they will, hopefully, tell you if the how matches the what, and if it doesn't where the two differ, i.e. where the the code doesn't do what you intended...

Reply on news.ycombinator.com