Writing Parquet files using Haskell | Hacker News Reader

Writing Parquet files using Haskell

datahaskell.org

89 pointscosmic_quanta33 comments

Looks like a pretty cool community building great tools with care. Using a functional language for data transforms sounds like a sane idea, haven't played around with it yet but it's definitely on my list now.

However they claim using Haskell for data science is "fast", which doesn't really mean anything until you have numbers to show. A little benchmark with pandas and polars wouldn't hurt I guess.

Haskell tends to be C-fast.

That's not what I see reported, they say Haskell tends to have bad memory layout generally and takes a 5x or so hit to performance.

Depends what you're doing, but it really can have C-comparable speed. [0]

Unoptimised/naive Haskell might be that slow. But that's true of a lot of languages and isn't particularly interesting to me. Java is slow if you do everything the naive way, too.

[0] https://entropicthoughts.com/on-competing-with-c-using-haske...

Java ain't slow even if you do really stupid stuff. Hell, I would even argue that java is the most resistant to stupid code. Some way over-abstracted everything linked data structure will be faster in java than it is in C.

Haskell's most badly optimised type, is the String.

Java's most badly optimised type, is the String.

Both of them need a string-builder pattern, the default operators don't work around things to do the right thing for you. They expect you to understand how data works.

The phrase “tends to” is doing a lot of work here. Inexperienced Haskellers tend to use the singly linked list for everything and that data structure just doesn’t perform in any language.

In many cases it's possible to write Haskell that runs as fast as a given C implementation. However the result will usually not be idiomatic Haskell.

This is underway: https://github.com/duckdblabs/db-benchmark/pull/180

Cool cool cool!

FWIW, https://duckdblabs.github.io/db-benchmark/ linked to from the readme, every result for "Haskell" shows OOM or undefined error. (Even on the large instance, even with the small input.

> Using a functional language for data transforms sounds like a sane idea, haven't played around with it yet but it's definitely on my list now.

It's been one of the plays that the F# community has also been trying to make (somewhat with a modicum of help from Microsoft's marketing arm, but not enough help from what I've seen) pitching F# as a language close enough to Python to feel familiar and useful to data science but with the performance help of rich ML types and the modern .NET performance ecosystem.

(To my experience: getting "fast" compared to Python seems easy for most functional languages. Getting data science out of Python seems hard for a lot of sociology reasons more than technical ones.)

In data engineering OO with method chaining is the standard for data transformation frameworks (Polars, PySpark).

I'd personally rather see Haskell become part of the options for https://arrow.apache.org/, but this is still a cool project.

I still love Haskell, had dedicated several years to it while at the university. Was impossible to land a job with Haskell. Did some Scala but it’s in demise. F# and OCaml are extremely niche. Ever fewer opportunities and engineering management is convinced it’s impossible to hire functional programmers.

In this new AI-driven world, is there still place for such a luxury as functional programming?

I mean few people still code by hand, few read the generated code, models aren’t trained on functional languages, it’s inefficient token wise to use functional languages - while a lot become self-proclaimed software engineers overnight by just prompting LLMs.

there are a lot of strong claims here, but to just pick one:

> inefficient token wise to use functional languages

where does that idea come from?

With LLMs, I'd expect if something appears more frequently in the training data, then the LLM would be more effecient or smarter at dealing with it somehow.

Surely there's way more content related to programming in Python, JavaScript, Go, etc. than in Haskell.

So you'd expect some things like: an LLM is likely able to come up with an approach that's suited to Python/etc., and an LLM is likely able to work its way through Python/etc.

My experience has been: when using an LLM with a nice language (Nickel-lang, a modular configuration language with types and contracts) that it frequently guesses slightly wrong as to what works (e.g. guessing wrong about how stuff like { x = x + 1 } would work) that it spends more tokens than it otherwise might.

Ooh, you're saying you need to ask it twice to get the same real-world result. I thought you meant the tokenization was less efficient (as if `Endofunctor` were tokenized `Endo`+`Functor` while `AbstractFactory` was a single token or something like that)

This is an interesting hypothesis to test. Depending on the embedding model, such less frequently encountered character sequences in the coding domain could indeed produce less optimal tokens. A dedicated model trained from scratch on Haskell would be optimal but wouldn’t generalize well beyond that, but can serve as a deliberately overfitted baseline.

What I meant was really about the output tokens, on the API pricing level of interaction with an LLM.

Interesting, most functional languages are quite terse and heavily whitespace, so you would expect raw token counts like that to be lower simply because source documents are on average shorter. Even having to make up for disadvantage things like C++ style operators like &&= all having single tokens from heavy use across a variety of languages but fancy functional operators like <=> being much more rare and in worst cases of undertrained models needing multiple tokens to express.

But yes, terser source documents and low raw token counts could mean lower prediction rates and more prediction attempts needed to get intended results (again, especially if the model was undertrained in that particular language).

Hiring for Haskell is very easy, unless you need to hire more than 100 people on the spot, or don't offer remote. I've been doing Haskell hiring for 10 years, with more than 50 interviews, the applicants are always great, each time we had to pick the best of multiple candidates.

Cannot confirm functional programming being inefficient token-wise, or worse at being generated than other paradigms. Claude is great at Haskell.

The difficult parts about Haskell, such as understanding type checker error messages, are gone thanks to LLMs.

Type-safe, side-effect free code degrades correctness a lot less under heavy LLM action in my experience.

I’m really happy to hear that. I wish Haskell were more widely adopted and more generally accepted as a solid enterprise choice without the bus factor. I can confirm that the applicants are usually great in every respect. I tried to introduce Haskell in multiple enterprises but failed at engineering managers’ reluctance to admit Haskell as an option - every time because of the bus factor argument. Scala used to be easy to advocate for as an alternative. To me, with LLMs there’s really no need to care about the choice of the runtime language unless there are SLOs to meet or library choices due to domain peculiarities.

As for token-wise inefficiency, this is based on my sporadic observations and discussions with friends and colleagues from the past three years. I can’t give you a fresh benchmark with latest models in a reproducible way. But I’m happily willing to accept your assertion at face value. This makes me curious to see for myself how the latest models perform, will perhaps set up a quick evals just out of curiosity.

From my past experience, the latest I’ve seen were Opus-4.6 and GPT-5.5 struggle a lot with standard GHC Haskell, no extensions, nothing fancy. Not that they produced impeccable Python or Rust. But it appeared to take several turns for obvious expressions, while at Java and TypeScript they were much better - fewer turns, time, and cost for the same verified results. I had assumed that the training set wasn’t large and diverse enough for Haskell and Scala, and this was the go-to explanation with everyone I talked about it. Generated F# code was good enough but not OCaml. With C++ there’s still quite some struggle, expectedly.

To me it resorted to the question, if we’re mostly generating code through LLMs now, which ones and what will it cost in total terms per task/capability completed. I’m wondering now if DeepSeek/MiMo or GPT-6 Luna can benefit from the terser, more (forgive the pun) load-bearing expressions.

For what it’s worth I recently decided to learn Haskell for fun and to keep my brain alive since work turned into a prompt factory.

Fable has made an excellent tutor. Maybe it’s just because the exercises I have it generating for me are relatively simple, but I have yet to catch it suggesting non functional or non compiling Haskell code.

There are companies like IOHK or Mercury that hired large swathes (50?) of Haskell programmers in relatively short time, so I don't think bus factor of the programmer market is an issue. Of course it's still an issue if a company hires 1 or 2 Haskell devs only, and those depart.

> from the past three years [..] the latest I’ve seen were Opus-4.6 and GPT-5.5 struggle a lot with standard GHC Haskell

If your experience is from years ago, I totally get that. For my standards, LLMs were pretty bad before 2026.

I haven't tried any serious LLM Haskell before 2026-02, because it was just not a timesaver.

I had good experiences with Opus-4.6 already on Haskell, such as it having detail knowledge on async exceptions, allowing it to immediately pinpoint difficult bugs that me and other experts missed, e.g. https://github.com/snoyberg/conduit/pull/530

But most of my experience is from Opus >= 4.8. I found it excellent at writing difficult TemplateHaskell and Generics implementations, such as oneshotting Postgres JsonPath queries derived from lens-like field accessors on Haskell types.

Note it's still not great at writing general high-quality "engineering" stuff, e.g. Opus 5.0 writes functions that refer to things out of their scope, such as:

    requireFooOr400 :: MyMonad m => Maybe Foo -> m Foo
    requireFooOr400 x = case x of
      Nothing -> throwHttp400 "Cannot perform [very specific action from the caller]: the inputs have no foo"
      Just foo -> return foo

However, that's not a lack of Haskell understanding, and it does that for all languages, e.g. in Python it keeps writing implementation details into API docstrings instead of function body comments.

> I haven't tried any serious LLM Haskell before 2026-02, because it was just not a timesaver.

Same, and now seven months later a GBP 20 per month Plus subscription gets me basically all the GPT-6-Luna-xhigh I can eat, which does a tremendous job of Haskell. It's even helped me with some really gnarly things like this, which require deep knowledge of the GHC RTS: https://github.com/tomjaguarpaw/bluefin/commit/c298733bdd67d...

It's possible it's much slower when working on Haskell than it is on other languages. I wouldn't know because I've only tried it on Haskell. But I'm very happy (and extremely impressed) by it.

Please note the "we had to pick the best of multiple candidates". Easy to hire haskellteers, but Haskell shops are not many. You would probably find more Clojure roles, but they are in decline.

My full time job is in scala and I find LLMs produce pretty solid code with it.

Reply on news.ycombinator.com