Hacker News Reader

Comment by treesurgeon | Hacker News Reader

Just mix nontrivial data structures with algorithms that iterate until numerical convergence. Team of experienced devs fought space leaks for weeks, project schedule crashed, directly caused by Haskell's lazy eval. Management and customers angry. Good times, good times.

View on HN

Sorry to be "that Haskell guy", but this kind of space leak is a solved problem. I call the technique that solves it "make invalid laziness unrepresentable"[1]. That doesn't mean it's a trivial effort to apply the technique to everywhere in your codebase that needs it, it doesn't mean that every library you use will have applied the technique[2], but it does mean it's a solved issue with a fix that follows from best practices. Space leaks don't take the insight of gurus to fix.

[1] http://h2.jaguarpaw.co.uk/posts/make-invalid-laziness-unrepr...

[2] Here's a space leak I fixed two years ago in a popular library by making invalid laziness unrepresentable https://github.com/mrkkrp/megaparsec/issues/486#issue-135418...

With all due respect. That is not a bug. That's like saying python slowness is a bug or that you can write undefined things in C is a bug. It's just the way the language works. Besides in the worst case you slap `StrictData` on the relevant modules. In the worst of the worst case you disable laziness altogether with `Strict` for the relevant modules. Frankly if an entire team of professional haskell devs are chasing a space leak for weeks then something is going very wrong and it's not the use of Haskell.

Complete rewrite in julia solved it.

Great! If you had such a special use case that you'd use hyper specialized programming language for numerics like Julia to solve why choose haskell at all? This confirms there was something very wrong in that development team. It's like having to write a web application and choosing CUDA.

For routine fast numerical algorithm dev, Julia is like a modern c++ or fortran. Hyperspecialilzed isn't the right adjective here.

Our team's big mistake was believing blog posts that swept laziness problems under the rug. In hindsight, there's plenty of criticism about Haskell's laziness in complex apps, and we should have taken that criticism more much seriously.

> Our team's big mistake was believing blog posts that swept laziness problems under the rug

I linked a blog post[1]. I wouldn't say it sweeps anything under the rug. In fact it takes laziness out from under the rug and shows you exactly how to deal with it. If you think that approach wouldn't have solved your problem then perhaps you could elaborate why.

[1] http://h2.jaguarpaw.co.uk/posts/make-invalid-laziness-unrepr...

That's great work and a terrific post.

We had lots of legacy subcomponents that permeated throughout the app's data structures, and as a practical matter, we couldn't reopen all that existing code. Some of that legacy code crossed organizational boundaries, so eliminating laziness throughout simply wasn't do-able.

So given all the above, there are two reasonable conclusions, which aren't contradictory:

1) If we could have reworked lots of existing code, we could have eliminated space leaks via @tome's approach.

2) Haskell's laziness is fundamentally misaligned with projects like ours, which has cross-organization code, long-running iterative algorithms, and complex data structures.

In desperation we used deepseq, which everyone knows is bad, and shortly thereafter we cut our losses by switching platforms.

Thanks! Although I'd be interested in debating the topic at length (and I also have a longstanding offer to resolve anyone's space leak problems) I don't want to be That Haskell Guy^{TM} who won't let the topic go, so I'll leave it up to you whether you want to continue the debate. I will say however, that this most recent comment is significantly less pessimistic than your original one (https://news.ycombinator.com/item?id=41170695) and I don't think I disagree with it (except if it subtly implies that Haskell must use laziness everywhere -- that's not true!).

It's great you found something that worked for you. For the spectators I think it's also worth reiterating that putting strictness annotations in data definitions that you didn't want to be lazy would also have solved it, just as rowanG077 suggested.

Reply on news.ycombinator.com