Hacker News Reader

Comment by maxander | Hacker News Reader

This sort of problem is Haskell's downfall, and very nearly negates all the other wonderful features of the language. It doesn't matter how fun it was to write or how the type system promised you up and down it was correct- if your program is taking up 20Gb of memory to run in simple cases it's going nowhere. And as this post shows, solving such problems is easily as laborious a process as tracking down memory leaks or buffer overflows in C; in these cases nearly all of Haskell's great and powerful abstractions are working against you.

The guys behind Haskell are utterly brilliant and seem to come up with profound new programming concepts on a regular basis, so I'm surprised they've not introduced fundamental features to solve or mitigate this problem. Until they do most people are going to be too scared to use Haskell for serious projects.

View on HN

As someone who's programmed extensively in Haskell, Java, and C++, I would say each of these languages have their own problems regarding memory. Write a decent sized Java application and you will have a ton of memory usage and perhaps space leaks as well. In a moderately complex C++ application, even the best C++ devs will have memory and space leaks.

I think the key here is that Java has JProfiler. C++ and C have Valgrind. Haskell has a heap profiler built right into the compiler. But Haskell doesn't have any specific tools for thunk memory profiling. The discussion in the community is now around building tools and automating this process. It's not like this is a fundamental issue.

Languages that have complex threading and locking also have to have more complex tools to debug these issues (ThreadScope in Haskell, VTune in C++).

What I'm trying to do is put these issues into context. It would be easy for a Python dev with the GIL to scoff at the complexities associated with C++ or Java threading and this is akin to what many do when looking at laziness. Laziness is a feature that comes with it's own types of bugs and we simply need tools to manage and track down those bugs.

The breakthrough with Neil's post is that even with current tools we now have a way to track thunk-based space leaks in Haskell and ideas for some more automation on the way.

> I'm surprised they've not introduced fundamental features to solve or mitigate this problem.

The problem is, to completely solve space leaks, Haskell should abandon pervasive laziness. But laziness is sold as one of the best things about Haskell, and most Haskellers do love laziness because it improves composability.

I myself hate pervasive laziness, and I'm all for removing it from the language. Unfortunately this argument will fail to convince most Haskellers because they're already sold on the bright sides of pervasive laziness.

Ok, this is all good, but from here there's one really annoying aspect of the Haskell community. They try to twist the reality to convince themselves. The typical reaction when these kinds of arguments are brought on the Haskell community is "space leaks are actually rare in the real world applications". Yes, this is the saying from the very community that emphasizes on program correctness. Sure, buffer overflow and memory leaks also rarely happen in other languages? Isn't "exact resource usage" also a part of correctness? At least in a broader sense?

Even SPJ (one of the designers of the language) admits laziness should retire now (it was a good experiment to develop purely functional ways, but now the disadvantages outweigh the advantages), but most Haskellers will never agree with this because laziness does also have advantages, and they love it. I actually gave up waiting for the Haskell community to abandon pervasive laziness, so just moved to other languages.

Sometimes I get a sense that many Haskellers treat hardware as merely implementation details.

What other languages did you move to?

Try Scheme aka Dr. Racket. It's up there as one of the "hacker" functional languages that everyone loves but no one really uses in production.

Scheme is deceptively simple though,.. so simple that you won't recognize it's beauty and uniqueness until you're really deep in. When I started learning scheme I quickly lost interest due to the fact that I felt I wasn't learning anything new.

It wasn't until I discovered sicp and had my mind blown again and again after every lecture did I realize how great lisps are. I recommend checking the course out: http://ocw.mit.edu/courses/electrical-engineering-and-comput...

Dr Racket is indeed very interesting, but I don't think it's in the same business as Haskell, i.e. statically typed "pure" functional languages.

I would have thought, if laziness-by-default was your main problem with Haskell, a proper alternative would be something like SML or OCaml, not a Lisp/Scheme.

Is F# lazy? A very quick glance seems to indicate that it is not.

It's strict by default with optional laziness.

Racket can be optionally strongly typed. That said, I'm with you on OCaml (or even F#) being a more natural alternative.

While it's not publicly available, there is a strict dialect of Haskell called Mu, owned by Standard Chartered. See https://www.youtube.com/watch?v=hgOzYZDrXL0 . It would be nice if the rest of us could play with it, but so it goes.

>Sometimes I get a sense that many Haskellers treat hardware as merely implementation details.

This is the ideal. The pinnacle of abstraction that HL languages strive for.

When you (inevitably) need to think about hardware in a extremely complicated system it means you hit a design flaw in the abstraction.

What a lot of haskellers don't realize is that laziness can be one aspect of this flaw.

> I actually gave up waiting for the Haskell community to abandon pervasive laziness, so just moved to other languages.

I certainly wouldn't hold my breath for Haskell abandoning pervasive laziness. It's a core principle of the language (Haskell is, after all, "being lazy with class"); if it were removed, the result would be Haskell in name only.

If you don't want pervasive laziness then using a different language is absolutely the correct thing to do. That's not a bad thing; it's just a case of having different tools for different jobs. Haskell may be the best fit for some projects but not others.

Completely agree. Laziness has too many benefits not to justify the added complexity of thunks and thunk-based space leaks. If you don't agree, don't use Haskell =)

> I myself hate pervasive laziness, and I'm all for removing it from the language.

If you want a ML-derivative without pervasive laziness, there are plenty out there (F#, OCaml, Alice ML and others.) It would make more sense to extend those with appropriate type-system features than try to redesign Haskell from the ground up without pervasive laziness.

> Even SPJ (one of the designers of the language) admits laziness should retire now (it was a good experiment to develop purely functional ways, but now the disadvantages outweigh the advantages)

Do you have a source for this? I recall him saying something similar but different in spirit.

Space leaks in Haskell are like memory leaks in a reference-counting garbage collector: a real problem, but one that is manageable. It does not mean we should abandon all hope and go back to exclusively manual memory management!

It's also a problem that tends to be exaggerated and is certainly not a reason to abandon laziness as others (not you) in this thread have suggested.

The equivalent of full-on garbage collection would be perfect, of course, and we'll get there eventually for non-strictness and meanwhile the existing system and tooling is improving all the time.

Finally, I think people also overestimate how much these things affect the popularity of a tool. I remember having pervasive, hard-to-track memory leaks in JavaScript just five or six years ago, and look how far it's spread by now!

While laziness may permeate a lot of the ecosystem, GHC 8.0.1 is likely to get better support for Strict: https://ghc.haskell.org/trac/ghc/wiki/StrictPragma

Woah, that looks great! Thanks for sharing this, I hadn't heard about the Strict pragma work.

This seems to be a tooling issue. Given the right hooks, it should be much easier to profile and debug this sort of issue than chasing memory leaks in other languages.

This is like trying to debug leaks in C with printf. It is just not done. Where's Haskell's valgrind-like tool?

There's quite a bit of tooling already:

http://book.realworldhaskell.org/read/profiling-and-optimiza...

https://wiki.haskell.org/ThreadScope

http://www.slideshare.net/tibbe/highperformance-haskell

Thanks for the links! I'll consume them as time allows.

The question then becomes, why couldn't the author use them to help pinpoint the source of his issues?

Neil Mitchell is a well-known Haskell expert, so I'm sure he would have used the available tools (if they were useful in the problem at hand).

When reasoning about Haskell code, due to non-strict evaluation you get a kind of "spooky action at a distance." If your program flow is something like "let result = C(B(A())", where A, B, and C are functions, an expression deep within function A may not be evaluated until you are in the middle of evaluating C -- or possibly never evaluated at all, if the value of that A expression isn't required to compute the return value of C. To reason about the runtime behaviour inside A, you may have to reason about the expressions that consume the return value of A. This can feel like chaos to someone who works with imperative languages, and who is used to nice things like explicit, linear stack traces: in Haskell, evaluation is driven by need, not by imperative instruction, and so the execution flow can look inside-out at times. That's not to say that better tooling isn't possible -- but it is a challenging problem.

> "spooky action at a distance."

I don't know about spooky, it makes more sense if you think in terms of "call by need".

Great questions! I think you're right this is a tooling issue. Haskell has valgrind-like heap/memory analysis built right in but not much for thunk-based memory allocation. Neil's article is really about Haskell's thunk analysis specifically and I imagine we will see some valgrind-for-thunk tools come out of this discussion.

Reply on news.ycombinator.com