Comment by chadaustin | Hacker News Reader

Comment by chadaustin | Hacker News Reader

Parent Full thread

Hi pron,

I get the feeling you're spouting very strong opinions without ever having actually written real production code in Haskell. Because what you're saying is nonsense.

Anything you can do in an imperative language you can do in Haskell - the only difference is that Haskell allows you to define subsets of IO that are enforced by the type system.

Mocking time is crucial for testing timeouts. You don't want your test to actually take five seconds. You want to rapidly test the code in the case that a timeout occurs before completion of the action and vice versa. You want those tests to be 100% deterministic and fast. With the World approach described you can get that.

Also, there is nothing about the technique described that limits you to unit tests. IMVU tests entire web services with this approach and it works well. "World" in that case includes APIs for accessing Redis, MySQL, the customer's session, time, concurrency, and so on. They're all perfectly faked out in tests, and the tests are 100% reliable.

Of course I haven't written production code in Haskell, and I'm not saying that there's stuff you can't do in Haskell.

So far, I've said two things. One is that the technique described in the post, which "uses the power of Haskell", is actually trivially done (and has been used by countless projects for years) in almost all imperative languages, and less intrusively (it doesn't require introducing new types and works with third-party code). I don't know much about Ruby and Python, but the JVM also allows you to define subsets of IO (even restricted by actual files) that are enforced by the runtime, or subsets of IO operations that will be enforced by the compiler. This technique has been used in imperative languages for a very long time. That's not an opinion but a fact.

The second thing I've said is an opinion, and is unrelated to a specific programming language. I said that a lot of very small, fully-faked unit-tests only get you so far, and it's fairly easy to get those tests to run 100% predictable. The more troublesome tests are those that aren't faked, and they are crucial to expose many bugs. That's it. I don't think I've said anoything too controversial.

Thanks for the clarification. But statements like this are why I think you're spouting crap:

"is actually trivially done (and has been used by countless projects for years) in almost all imperative languages, and less intrusively"

Is simply not true. In a JavaScript test framework, drawing from top-of-mind, recent examples, you cannot prevent the code under test from replacing document.location.

In Java, as you gave in another examples in this thread, you can't statically prevent access to the RNG or perhaps the current time.

In Python, there's nothing stopping any code from calling into a C module that has arbitrary effects.

Haskell is unique among any languages I've ever used in production that allows deterministic testability as a static, enforced-by-the-compiler guarantee.

I partially agree with your second comment, and I do think other forms of testing (such as fuzz testing, integration tests with third parties, acceptance tests) are valuable, but the technique described can be used in much larger tests than simple unit tests. As I mentioned, IMVU uses it for testing entire web service request handlers.

I get the feeling some people think that World is onerous and expensive to set up, but in practice you just write all of your code in World and it's not boilerplatey or complicated at all. The framework around World is set up once and never touched again.

* Disclaimer: I no longer work at IMVU, but I super super super miss Haskell in production.

I don't know too much about JS or python (I'm mostly C/C++/Java/Clojure/Matlab) but there's nothing to stop you from accessing C, in Haskell either, nor from using unsafePerformIO, and while in Java you can't prevent access to random numbers etc. statically, but you can certainly do it at runtime (though, in Java, too, you can call C and then all bets are off).

> Haskell is unique among any languages I've ever used in production that allows deterministic testability as a static, enforced-by-the-compiler guarantee.

Oh, that's true. It's just that if your tests are guaranteed to be deterministic, I don't see why the fact that the property is enforced statically is so important.

Personally, I think Haskell sacrifices a lot for the sake of static guarantees (including the ability to reason about programs not before they run, but during and after). Also, its reliance on Curry-Howard for verification is limited and narrow-sighted, and its insistence on referential transparency everywhere is interesting yet misguided. I'm now preparing a talk for the Curry On/ECOOP conference where I'll show -- among other things -- how all monads can be easily translated to imperative constructs (I won't go into details here, but what you need is continuations and continuation-global variables) in a way that makes them more understandable and compose more naturally than in PFP (i.e. there's no need for monad transformers). Even though this model does not equate the subroutine with the mathematical function -- as PFP does -- it is no less verifiable than the PFP approach. But I digress...

Thanks for your response! Would you be open to continuing this discussion over email?

I'm caustin@gmail.com

When are you giving this talk? I am very interested.

July 6th

You can prevent access to `unsafePerformIO` and other unsafe language features using "Safe" Haskell (e.g. -XSafe). This happens at compile time. Functions like unsafePerformIO (IO a -> a), pure C FFI functions, overlapping instances,and TemplateHaskell are not allowed. Nothing is stopping you from accessing C but you can force FFI function to reside in IO.

Haskell lets us have generic "mocks", which is -way- better than any kind of expectation-based mock. (admittedly, the example in andy's blog post doesn't show the power of this technique very well)

But my experience with mocks is that expectation-based mocks are fragile and make it easy to write tests that pass but are fundamentally wrong; if you just say you expect XYZ to come out of the database, even if your code gets refactored to not insert XYZ into the database earlier, the test will pass despite the code being completely broken.

With the haskell World approach, you don't have that - the fake database is fake because it's in memory (and thus very fast and uses a blank state for each test), but it's actually implementing the same semantics as the real database.

Some of the other IO stuff we've encapsulated is just expectation-based mocks and as you said that's not new, but it's still clean and doesn't require any changes to the code under test to make it mockable, which we did when we used mocks in our PHP.

How do you verify that your mock behaves exactly as the real database? The type system certainly helps with function signatures, but what about behavior?

As with any fake implementation of a service, you definitely need coverage that verifies that the fake and real have the same behavior.

In-memory implementations are very useful, especially for larger tests, but, again, I don't see anything in Haskell that makes it easier. For example, on the JVM tests often use Google's Jimfs[1] in-memory file system (the JVM has pluggable file systems implemented as Java interfaces), and for relational databases they use H2[2]. None of this requires any code changes, perhaps thanks to the JVM's extreme pluggability. The JVM uses SPIs for almost everything, from encryption to filesystems to sockets -- you can fake the entire OS (almost) in Java, and what you can't fake easily, you can still fake with an agent (unrelated, but starting in Java 9, even the JIT is pluggable, so you can control machine-code generation at any granularity you want). Just since my conversation here with chadaustin, I've written a small library that uses an agent to make the entire JVM use a virtual clock[3] -- again, with no code changes necessary in the tested code.

[1]: https://github.com/google/jimfs

[2]: http://www.h2database.com/html/main.html

[3]: https://github.com/puniverse/timewarp

Reply on news.ycombinator.com