Functional Programs Are Imperative Pumpkins | Hacker News Reader

Functional Programs Are Imperative Pumpkins

blog.eventsourcing.com

56 pointsyrashk72 comments

What does the word "pumpkins" mean in this context?

I think it is an allusion to the story of Cinderella [0], some versions of which have her carriage transforming (back) into a pumpkin at midnight.

I see the implication being that when all is said and done, functional programs are really just imperative programs in disguise (I don't have an opinion on the matter).

[0] https://en.wikipedia.org/wiki/Cinderella#Cendrillon.2C_by_Pe...

Yes, that's the allusion!

The implication was that rather systems composed of functional programs but non-functional databases are not functional (i.e. the functional components are dwarfed by the imperative nature of mainstream databases)

This pumpkin analogy goes away when you consider a a functional database is one where only writes and reads happen (no updates). Indeed this is what functional means: And object is distinct from its state because, in the real world, we do not consider what something is to be the same as what happens to it. Imperative programming locks the two together inseparably. Imperative is where we are merely the accretion of what happens to us. We have no history and no future.

Isn't this article just stating the obvious fact that programs at some point do I/O, so no program can be purely functional?

Every Haskell tutorial seems to introduce the IO Monad to illustrate this distinction.

While not new, I find the discussion around immutable storage important and interesting. Datomic[1] comes to mind. There are many scenarios and applications, where it would be nice to just write things to a structured log and build an application on that log.

Unfortunately there are some roadblocks, like little information and conversation about the topic and no real mainstream implementation of these ideas.

I've been working on a system, that has is based on an immutable data layer and it is just a wonderful thing to work with, as long as you have enough disk space.

[1] http://www.datomic.com/

I'm curious, how is this different from using an event oriented database, or using your normal DB but changing the schema to be oriented around immutable events?

Now I trying to use Event Sourcing (mainly for sync) for a "normal" invoice app, I find several problems. The thing is that the event log is good for write but awful for read. Re-compute everything is problematic and the true is 90% of the time you want the last version, not all the history. Also, the split in logic from the app / engine is stupid. I think is good if we move past the idea that the DB engine is more dumb than a rock, and embrace it. But current tools are not ideal (sadly, I know what is live like this: I have used Visual FoxPro!)

Where I think this could lead to something great is if the DB is alike this:

- Relational

- You have normal tables. Consider them your up-to-date cache. Most scenarios will be covered here (ie: The log recomputing is NOT at the app level, but at the DB level).

- You have a event log where everything is stored, plus your "tables" for the info you need up-to-date. But you don't need all to be a table if not make sense, and do full re-computation if desired.

- Your backup, sync and load history is around the event log.

- You don't even need a "table" if you wanna only a "index", this lead to me:

- You submit a data request (like a POST), and the DB convert it to the tables and log, but:

- You can instead (or also!) hook here and build secondary indexes and do other stuff. You can do it async, put the computation in a queue, do validations, etc as fit.

So, what is not here and need tooling is marry the "normal tables" + "event log" + "logic that route this", so putting:

Data -> WAL -> Router -> Event Log | Tables | Index | ping external tools

And

Request Data -> Router -> Get it from:Event Log | Tables | Index | ping external tools

Have you seen PipelineDB? It's Postgres, but with something called "continuous views", which are like regular views except they're cached and updated on writes, not re-calculated each time they're read. Sounds like one could use plain tables as the "event log", and these views as the up-to-date cache. And aggregation queries work too, so your regular table can be more than just a cache.

I'm not affiliated, not even a user, it just sounded interesting when it was submitted here.

I have not use it, I my first glance make me think is only about analytics, but maybe that could work. I will try and see how well...

Jeff from PipelineDB here. Yes, PipelineDB is an analytics product designed to continuously query high volumes of streaming data. The top use cases we see are building realtime reporting dashboards and realtime monitoring and alerting systems.

PipelineDB excels at workloads where you know the queries you want to run ahead of time, where your workload fits within the confines of SQL, and where there is a high degree of distillation (aggregations, sliding windows, etc.). We have large customers including Charter Cable / Time Warner Cable, MediaMath, SmartNews, MOAT Analytics, Cradlepoint Networks, Paddy Power Betfair, and others using the system at scale and we offer a clustered edition of PipelineDB under a commercial license, but PipelineDB's single server edition is open-source.

We have a live chat room here for technical questions:

https://gitter.im/pipelinedb/pipelinedb

You are quite right, event sourcing on its own is not great for read, and the "re-compute everything" (aka foldl) is problematic. CQRS solves this by producing a read-side that aggregates events into your domain. Classic event sourcing also attempts to solve the re-computation problem by snapshotting.

It is great that you mention that 90% of the time you want "the last version". That was one of my discoveries with event sourcing as well.

So what I came to is that re-computation (foldl) is a nice generalization, but isn't very practical. Instead, index the events and build the state on demand by querying the indices. This way, [plurality enabling] domain modelling moves to the code (as opposed to the database) and allows for quick changes in how you see the information.

You can read more about it at https://blog.eventsourcing.com/why-use-eventsourcing-databas... and https://blog.eventsourcing.com/lazy-event-sourcing-ed7e59007... as well as other articles on that blog.

The cognitive struggle I have is how not only 90% on reads I need the last version, but also on write.

The classical example is how avoid duplicated data or do validation across tables. One of the requirements of my app is deny a sale if the customer have pending invoices. The tricky part with invoice is that entry must be fast (I'm competing against DOS apps that fly at data-entry).

So when ES say "Record the event, then build the read side at later time", instead me (and I think, many of us) wanna the opposite "Put into the event log after we have do validations".

Of course, some actions can be delayed, but some need be as-now. How do this elegantly?

My initial attempt is code against the tables and use triggers for recording the story, but that is not as nice...

This is actually how I do event sourcing in many ways. The entry happens through commands, which can do validations prior to emitting a stream of events to be journaled. I also expose a locking mechanism to commands so that they can ensure the uniqueness or property validations are correct.

In lazy event sourcing, you don't "build the read side at a later time", it's built upon receipt — in a form of indices, as they are THE read side and you pull this data and form domain objects on demand (by querying your indices), just transforming one representation (relevant events) into another (domain objects).

Does this help?

How you "expose a locking mechanism to commands"?

And how you build the indices?

Offtopic to the OP but related to Datomic.

I saw Rich Hickey's talk about Datomic recently and immediately thought it would be ideal as part of a distributed runtime. Without having actually programmed in Erlang, but having seen plenty of Joe Armstrong's talks, I understand that the point of replicating data in messages rather than passing around pointers is so that if a machine dies, the data isn't lost. But if you built a distributed runtime that used Datomic as a distributed heap on which you allocate large amounts of data which you expect to pass around a lot, you could instead pass around only pointers to the data and not worry about data loss, and save big time on message throughput. Does anybody know if something like this already exists?

I feel that one of the biggest misconceptions about monads and IO in Haskell is that they are an "escape hatch" in the language that allows you to perform imperative actions. This is plainly false. Monads are just another functional programming idiom, defined by two regular functions, `return` and `(>>=)`.

What monads allow you to do is have a purely functional model to represent IO, among a great many other things. Haskell is, in a sense, the best imperative programming language ever created, because it lets you manipulate imperative statements in a purely functional manner. You can stick statements in data structures, compose them, replicate them, generate them and only evaluate them when you want to.

I agree. But this is a misconception about functional programming in general, that its purpose is to somehow replace imperative actions that are done by the computer system with something else which is not imperative.

But that's not the goal. Functional programming is there to provide a better formalism for writing programs, it doesn't want to change the way the computers actually execute the programs (or how the systems eventually internally work).

That means more work for the compilers to figure out how to make the same old imperative program out of a functional one, which is a good thing (it's generally the direction the compilers evolve).

Benefits of functional programming are not in how programs are executed (quite to the contrary - it's actually more effort to make them as efficient as good imperative programs), but in how the programs are written by people. I think the article is a little confused about this.

For another example to see this, even notion of non-pure function goes already in this direction. Today, having lots of (small) functions seems to have negligible downsides, but historically this wasn't the case at all, look at any better program that had to fit into memory of an 8-bit computer. These were rarely written in this style, because compilers were simply not capable enough to produce a reasonably compact spaghetti machine code out of those (by means of inlining, for instance).

To sum up: harder to write, harder to compile, harder to make efficient. Hum... difficult to qualify it as 'better' after this.

I agree with you. Yet, strangely enough, it is better. Maybe we could say, easier to read, easier to modify, easier to reuse?

Let me return to that last example. Take a program written in procedural style, and an identical program written in a big ball of mud style (all variables are global, one big control flow). This is a discussion from like 1960s. It's harder to write for human, and harder for compiler to compile and make efficient program in procedural style.

Yet, I am sure today, you would prefer procedural style. It makes sense (trust me, I actually work with big-ball-of-mud programs written 30+ years ago at work, AMA). I don't care what work the compiler has to do, I care about my own time.

Functional programming is just another step on that road.

As a side note, I really think people who prefer FP should avoid (over)selling it as more computationally efficient approach, because it still isn't (even though it probably will be in the future). It's a little bit of optimistic dishonesty that IMHO hurts adoption of FP and feeds the misconception that I talk about.

Harder to make errors, easier to modify.

But what you said is true too.

Monad is not itself an escape hatch, but Haskell's escape hatches are mostly monads (ST, IO, etc). Monads may be a functional programming idiom, but monads in Haskell are deeply magical. So "monad = escape hatch" has some basis in reality.

It's important to distinguish between imperative code and IO. Many pure algorithms are most naturally expressed imperatively. Viewed in this way, Haskell is actually pretty bad at imperative programming.

Myers Diff is an example. See diffST2 at https://hackage.haskell.org/package/uni-util-2.3.0.1/docs/sr... for the Haskell version. Notice how simple expressions in Myers's paper, such as `V[k−1] < V[k+1]`, explode into ugly things:

    vkplus <- readArray v (k+1)
    vkminus <- readArray v (k-1)
    (x,l0) <- if vkminus < vkplus...

Here we've paid a syntax tax for mixing the "imperative" array-read with the "pure" value comparison. Similarly, notice how the 10-line recursive `scan` is a less-readable implementation of a one-line while loop in the Myers paper.

So in practice, Haskell is rather painful to use as an imperative programming language.

Technically the escape hatches are only the stuff here: https://hackage.haskell.org/package/ghc-prim-0.5.0.0/docs/GH...

So you could write

  case readIntArray# v (k +# 1#) realWorld# of (# vkp, _ #) ->
  case readIntArray# v (k -# 1#) realWorld# of (# vkm, _ #) ->
  case vkp <# vkm of
    0# -> ...
    1# -> ...

with syntax sugar reasonably close to V ! (k+1) < V ! (k-1).

The problem is the state tokens; I ignored them by using realWorld# and _ above, but that makes the code not referentially transparent. So in practice, Haskell always passes them around in a monad (Except in all the core libraries, which use things like accursedUnutterablePerformIO to go back down to the thunk level: https://github.com/haskell/bytestring/blob/master/Data/ByteS...). But reads are (mostly) pure so it shouldn't break anything to provide a pure interface.

Alternatively, the verbosity can be seen as a syntax problem, with the corresponding solution of a better do-notation: https://www.reddit.com/r/haskell/comments/4dsb2b/thoughts_on...

so you'd get something like

    (x,l0) <- if (<- v ! (k-1)) < (<- v ! (k+1)) ...

(again using ! for readArray, which has some impracticalities)

That's still not quite as good as C, it really needs the notion of sequence points: https://en.wikipedia.org/wiki/Sequence_point

Monads are just another functional programming idiom, defined by two regular functions, `return` and `(>>=)`.

But in terms of I/O with the outside world, monadic patterns become useful in Haskell because the type of your main function is also `IO ()`. That's the "escape hatch", not the use of monads in itself.

Everyone knows the real escape hatch is unsafePerformIO!

  no program can be purely functional

Pure functional programming means building up a machine. This machine will eventually be executed by the runtime system (at which point IO will happen) but from the programmer's perspective a program can still be purely functional.

By this I mean that a function that takes an Int and outputs an Int will have type `Int -> Int`, and a function that takes an Int, does logging, and outputs an Int will have type `Int -> IO Int`. Both are purely functional -- they're pure transformations from an Int to some other thing. For the first function that other thing is another Int, for the second that other thing is an action that can be performed by the machine which will return an Int.

So from the perspective that matters to us a developers programs can be purely functional.

(This becomes even more useful with something like PureScript's granular effect tracking https://github.com/purescript/purescript/wiki/Differences-fr...)

Yes, in Haskell monadic functions are strictly speaking still pure functions, but it's the set of mutations performed by the package of runtime + your code that matters.

Do-notation shows that there's a trivial mapping between monadic code and imperative code. For all practical intents and purposes, things happening in the IO Monad are not 'functional', because your mental model needs to consider the mutations they represent.

What functional programming encourages you to do (as opposed to OOP) is segregate the pure aspects of your program, under the theory that pure code is easier to write, maintain, and test.

Effect segregation (a more appropriate term than “purity”) is a matter of using types, and has nothing to do with functional vs. OOP.

The type system has a huge impact on how easily you can do effect segregation. Haskell has a type system that enforces effect segregation for you. Java doesn't.

Classic OOP encourages modeling your problem as a collection of inherently stateful objects, making effect segregation more difficult.

Of course, you can write in a functional style in Java, or an imperative OOP style in Haskell, but the language isn't going to help you as much.

There's no law of nature preventing you or anyone else from designing an object-oriented language that enforces effect segregation.

Even if such a programming language existed, does using both paradigms in the same program mesh very well?

Effect segregation isn't a “paradigm”: it's just having the right type structure to classify programs according to their possible effects. And, sure, it's compatible with object-orientation. Why wouldn't it be? You just slap an effect annotation (explicit or inferred) onto every method signature.

Pure object-orientation is a bad idea for other reasons (e.g., the binary method problem), but “inability to control effects” is totally bullshit.

Of course you can design a type system that has objects and also segregates effects. The question is whether that would be particularly helpful for structuring a program if you're trying to write explicitly in the OOP paradigm.

Functional programming is a paradigm, and it emphasizes the segregation of effects. OOP traditionally does not.

You're conflating paradigms (program design philosophies) with language features (program design tools):

(0) Functional programming is a paradigm that encourages expressing computation as expression evaluation, and discourages relying on more computational effects than are necessary to obtain a desired result.

(1) Object-oriented programming is a paradigm that discourages directly manipulating data structures from different parts of a program. Objects expose operations that act on a data structure hidden from the rest of the program.

(2) Effect segregation is a feature of some programming languages, notably Haskell and its descendants. An effect system classifies computations according to their possible effects, just like a type system classifies computations according to their possible results. You wouldn't call static typing a paradigm, right?

You're right, effect segregation is not a paradigm per se.

The fact still stands that it's closely aligned with the functional programming paradigm, and not OOP. Programing paradigms and language features are closely related: certain features make certain paradigms easier, and paradigms spur the development of particular features.

No, that's not exactly what he is saying. He's saying we write logic in pure functions, but then save the data in a mutable database. But he's saying we don't have to -- with event sourcing, even the data we store is immutable, so more of the system can be purely functional.

All but benchmarks.

>Isn't this article just stating the obvious fact that programs at some point do I/O, so no program can be purely functional?

I agree with the point that a program can't be purely functional, but lately I've been wondering if you could still have a programming language that is purely functional (preferably also total). It is designed for writing data transformation libraries and needs to be called into by a stateful language that handles IO and orchestration. If you've seen Gary Bernhardt's talk Boundariers [1], this language would be the purely functional core.

[1] https://www.destroyallsoftware.com/talks/boundaries

This isn't directly related, but if this person wants FP + Java ecosystem, why not use Clojure or Scala?

There are functional options for almost all of the big ecosystems now.

Probably because as the author mentions, they weren't trying to rationally decide on a language/ecosystem, but looking to rationalize the decision they already made.

I mean, there are tons of features in Haskell that one misses when using Scala. The type system isn't as useful, Scala's version of ADTs (case classes) are really inconvenient to work with, different typeclass system, etc. Can't speak for clojure.

I know that Clojure has its warts, but it seems to have a better reputation in the functional community than Scala. Scala seems to want to support all paradigms (or many paradigms simultaneously), whereas Clojure was created to be more tuned to FP.

The difference between Clojure and Scala is that Clojure is a dynamically typed LISP whereas Scala is statically typed. In the FP community, in general, statically typed languages have a better reputation and are considered to be more tuned to FP. Yes, Scala mixes OOP and FP but OCaml(famous FP language if you've never heard of it) also does exactly that.

I wouldn't agree to that. LISPs were the first Functional Languages as FP has its roots in Lambda Calculus. Scala is much more multi-paradigm, and I think its further away from the FP languages. Even Wikipedia agrees: "An interesting case is that of Scala[28] – it is frequently written in a functional style, but the presence of side effects and mutable state place it in a grey area between imperative and functional languages."

A lot of FP practitioners tend to come from academia and most of the research currently lies in provability, so a lot of the FP community seems interested in strongly typed languages, because they overlap the language research communities, but I doubt anyone in the community would claim they are more tuned to FP, because that's just a lie. In fact, a lot of the challenge of typed FP languages is designing a type system powerful enough to express all types of functions, so they're often more restrictive in how they can implement FP.

https://en.wikipedia.org/wiki/Functional_programming

Yes, Scala is multi-paradigm and it support OOP as well as imperative programming but the community is very much dedicated to FP. All I see is typed FP that expands my mind.

A lot of lisp people get testy when you call lisp a functional language.

It goes the other way too though.

Records, for example.

I think the essence of the point was made for me by reading "C is Purely Functional" http://conal.net/blog/posts/the-c-language-is-purely-functio...

Functional and imperative are paradigms that can "encode" the other, so to speak.

Appel's classic "SSA is functional programming" offers another interesting way to look at this equivalence.

I don't get this. Any program that interacts with the outside world isn't "pure", by this definition. What's wrong with using monads? With monads, you can still do all those side-effecting operations, but be encouraged to separate out the pure bits.

From the article: The "Land of purity" lies above the "Dirty sea of data" .

Wouldn't "Dirty sea of state" be more appropriate?

Perhaps!

Why do people, even obviously skilled and well-read people, conflate functional programming with pure programming?

You so not need to have pure code to have functional code. You don't even need to have pure code to use a monadic approach.

What do you think "Functional Programming" means? Obviously people and various groups have their own made up definitions, but a commonly agreed upon definition in the PL community might be "Programming with Functions". Functions, by definition, are pure.

> Functions, by definition, are pure.

That's just not true in our domain. While it may be true in some categories, we do not program in an abstract category. In our domain, "pure" functions are distinct enough that we call them "pure" functions and even in Haskell we draw a line between effectful code and pure code. As a core concept in modern programming, the word "function" makes no judgement about how many side effects are there.

I'm not sure why people think otherwise. Or perhaps all Lisp and OCaml programmers are just liars and cheats.

EDIT: I just had a long conversation twitter about this and I'd like to offer an example of a functional style we commonly use even with side effects: tail recursive loops. We often use this modeling technique even in handling I/O because it dramatically simplifies the logic for consuming a stream.

>> Functions, by definition, are pure. >That's just not true.

I think there's some naming confusion here. That's because programmers overloaded the original meaning of "function" so that it's equivalent to "procedure". So now when something behaves like an actual mathematical function, we have to explicitly call it "pure".

PS. i remember seeing you in some Yogscast videos, used to be a huge fan of those. funny to meet you like this.

It's perhaps confusing because people think the constructs they read about in category theory have 1:1 mappings to the world of programming and they don't. It's why we talk about the category Hask.

Every single concept is overloaded.

P.S., Wow. Someone here who knew about my hobby? I figured it'd never happen.

> people think the constructs they read about in category theory have 1:1 mappings to the world of programming and they don't.

Of course they don't. But they can still be considered a useful model for programming in the same sense that Newtonian physics is a useful model of our world.

Of course, but I still think it's important to remind people that functions need not be pure to invoke the benefits of functional programming.

The article in question acts like it is a big surprise gotcha that functional programs have imperative elements. This shouldn't be a surprise to anyone and in fact it's a feature! The accessibility of that feature is often a major component of what languages are judged on.

This isn't a point of semantics, it's definitely the case that we use functional techniques in an imperative paradigm to great success.

Thanks for the feedback!

It's not that functional programs have imperative elements or not. That's kind of besides the point.

The point is that putting functional components into workflows that deal with arbitrarily mutable data is reducing the potential we have with the functional model.

The article is simply an open-ended invitation for a discussion on how to do this (and it spark a little bit of that conversation) with some of the proposed options on the same blog.

We're talking about definitions, so you may continue to refer to 'functions' however you like, but from a functional programming perspective, I'm going to posit that it is useful to distinguish between functions and 'impure' functions. :)

I think your position is that because there is an escape hatch in languages like Haskell, that negates the usefulness of equational reasoning. I'll argue this isn't true, in the same way it's useful to pretend ⊥ doesn't exist and that we don't bork our unbound recurison or whatnot. Once we do that, we now can reason about our functions as if they exist as arrows in a cartesian closed category (of types). This is incredibly useful. We just have to restrict ourselves to a subset of the language. We still get that same benefit even if part of our program call impure functions. This is the reason we want to model as many side effects as we can in the types (an IO monad or Task in Scala).

I saw your twitter convo, I don't know what you mean by this:

>"If we define functions as pure, then functional programming doesn't exist outside of extreme specialist applications."

All kinds of Haskell applications are built with side effects. Are you under the impression UnsafePerformIO is the only way to interact with the world in Haskell?

Why has this been upvoted? The author is a novice and provides no insight. This just reads like he finally got to the second page of your standard book on functional programming and somehow came to believe that everyone else is still on page one. What village is this guy from?

Probably because event-sourcing is beginning to trend.

The article is not that interesting in itself, but the conclusion is a good starting point for a conversation that may be.

> Systems are largely not functional. But they can be.

So, how?

My research into this topic has led me to lazy event sourcing so far https://blog.eventsourcing.com/lazy-event-sourcing-ed7e59007...

My current view is that by eliminating or reducing the need to serialize domain objects, we're creating a system which is easier to argue about and provides an easy way to adjust domain models as the understanding or circumstances evolve.

Yeah, now this is a nice continuation to the conversation :)

I too have been mildly obsessed with the kind of thinking you're developing in your post - very nice article!

I've been very attracted to event-sourcing since I realized that now that React and redux have mostly solved the UI problem what remains to be solved is the problem of how to store our beautiful event-driven data.

EDIT: re-read your comment, felt the need to say that your "current view" is exactly mine too

I am happy to continue this conversation off-HN, perhaps Gitter? http://gitter.im/eventsourcing/eventsourcing

> What village is this guy from?

I know this thread is about functional programming and so condescension is the norm, but isn't it about time we change that part of our culture a bit?

You offer significantly fewer insights than the OP. Zero arguments, just name calling. The author reads this thread and is a real person. Are you proud?

I, for one, found the article an insightful articulation of things I kinda sorta already knew but hadn't acted on. Also our team is in the middle of building an Elixir backend with a classically designed mutable DB so the article helps me reflect. Maybe now is the better time to change this.

Elixir has more to offer than just a functional approach; there is OTP, lightweight processes, etc. If these are things you need, Elixir might still be an excellent choice.

I meant that I'm now considering changing the thing the article is about: using immutable data in the database and considering any mutable parts of it a cache, in a sense. Basically he's describing CQRS but not using the term.

Moving off of Elixir certainly isn't being considered :-) We chose it particularly for its concurrency and robustness features, which are important to us (we're a messaging component for marketplace/platform sites so deliverability and concurrency are very high on the list). We'd have picked Erlang if Elixir didn't exist.

I am indeed describing CQRS, or at least the way I see it. The way I am doing is this by making my read side a combination of lightweight "code-only" domain models and heavy db-side indexing of events themselves (as opposed to domain model tables of some sort)

If you have any thoughts or questions, feel free to reach out to me (my nickname is the same almost anywhere, twitter, gmail, etc.) or Eventsourcing gitter room http://gitter.im/eventsourcing/eventsourcing and I'll be happy to talk about these things.

I liked Elixir, other than it wasn't statically typed, but I found the ecosystem quite young which means you will have to spend a bunch of time purely keeping your code up to date with libraries as they change and if your tech integration doesn't exist, you are on your own.

After a bit of consideration, I figured F# makes more sense for me. It has a large ecosystem covering many technologies, its all moving to open source / any platform. It also has lightweight processes, and lots of tooling.

I agree that there's little point in using Elixir currently if you don't need its concurrency/robustness story.

Reply on news.ycombinator.com