Functional Core, Imperative Shell (2012)
destroyallsoftware.com
destroyallsoftware.com
I find both of these scenarios disappointing and frustrating, because technology (and software) is usually about improving processes and systems so better results can be made more easily. But at this point it feels like a lost cause, or rather a hill not worth dying on. For my own projects, I go this path; for others, I work dumber and keep the corporate apple cart upright.
Rant aside, there is a video which I really like that demonstrates this functional core/imperative shell approach: Solving Problems the Clojure Way - Rafal Dittwald. It is a Clojure talk, but it gives an imperative to functional refactor demo in JavaScript in the middle of the talk. Even though it is a toy example, it is big enough that the result shows a meaningful difference in approach.
Without having a certain number of key developers pushing the transition it's hard to make this jump.
I would also say that this kind of transition can be had progressively, I myself know a person who pushed a company into a fully-typed static pure functional programming a team that started their whole business in php. Gradually moved them into typed functional php and then into Haskell. He was lucky enough that the team/company was small enough for that at a bigger scale this would've been much more difficult.
In other places I've seen the transition happen (into the functional typescript ecosystem which is a sweet spot imho) it has been mostly the CTO pushing for it.
I know they moved into a more functional php, then typed php, then started rewriting some parts of the system to Haskell. The company got eventually bought up by a competitor and killed.
Few companies I know here in Italy are going through a similar into more functional and statically typed php with efforts such as facile.it with libraries like that:
https://github.com/facile-it/php-codec'
Most go through the fp-ts/effect-ts ecosystems and never jump on the scala/haskell languages as they find functional typescript to be the sweet spot for them.
On refactoring it's a bit easier, especially since if you choose the right scenario (big new feature), you can just do it that way and then try to teach by example. It's less scary that way, even though there still is initial resistance. Also I think it's better this way, because when you write the tests for the new feature, you can show them side by side with other tests in the codebase. The simplicity of the new tests, and the fewer lines of code but higher coverage, helps sell the deal.
The real key aspect of this approach is that your state is not spread around the code and changing all over the place, but instead a big State -> State function built out of many smaller referentially transparent functions. (i.e. the complete opposite of OO programming, which IMO is a generally terrible idea that only works well in Smalltalk or in distributed systems).
You can achieve a similar outcome in the procedural setting with a "double buffered" state. Your procedural function writes out a new state based on the previous state, and the "imperative shell" only ever touches the "completed" states, and then flips them so the current state becomes the previous state and the previous state becomes the buffer to have the new state written into.
Less convenient but if you need to have tight control over memory allocations or performance this can be beneficial.
Or a draft/publish workflow where you’re constantly re-use memory instead of remembering published documents.
I like it! It’s clever and seems to be generally applicable. Is there a name for it?
. take a performance hit
. somehow track/update costly intermediaries in your global state, and this can balloon out of control really fast
. pair it with some memoization mechanism/framework that automatically manages "derived state" variables
The same is true of the functional core idea really, you're still going to have a big tree of function calls rebuilding things unless you also add some memoization to it. React and friends do it because DOM manipulation is very slow.
But think of IMGUI style GUIs, they're rebuilding the entire user interface every frame and yet are often more efficient than many retained mode UIs. Redundant computation is not as bad as it sounds at first glance.
https://github.com/hoplon/javelin
It's sort of a version of option 3
I don’t know what metric you use for that, but they are definitely not more energy efficient — retained mode is a must for any battery-powered device.
If you're rebuilding things a lot anyway, it's actually more efficient to just accept that rather than work around it.
Though speaking of immediate mode UIs, I think a lot of the problem there is that most of the immediate mode UI libraries assume they're being used as the user interface of a Game or 3D render application which needs to render a complex 3D scene regardless, so who cares about the extra 400 microseconds the UI takes to render.
You could imagine optimizing immediate mode UIs quite a bit for energy efficiency by not redrawing anything if there's no use input, or selectively redrawing dirty areas where it makes sense.
That’s pretty much what a retained-mode UI does.
No need for a retained data structure to implement the above.
You can often optimize a single, large state transformation by utilizing that it does a lot of similiar work. You can also often get a big performance boost by batching up computations.
Half of all performance problems are solved by introducing a cache. The other half is solved by removing one.
I have been thinking about and trying to work with this paradigm for a few years now. To be honest, it's been a struggle. Partly because it requires unlearning some habits and partly because there's limited opportunity to practice in my $DAYJOB where I'm working in codebases heavily invested in other paradigms.
I've seen and read many, many videos, blog posts, etc. on this topic. Everyone loves to introduce the concept with a contrived example and then stop there. What I would love to see more of are complete examples for everything from simple to complex applications.
I think there's potentially a lot of value in thinking this way (or at least taking as much from it as you can), but it's hard to learn and then to teach others without more examples to draw on.
In practice my imperative shell tends to have two layers. The inner layer is responsible for executing the imperative logic, while the outer layer is responsible for initialising configuration and dependencies, invoking the inner layer, and adhering to any sort of external interface that it may need to satisfy. Everything from the inner layer down through the functional core can be comprehensively tested using stub objects only -- no need to patch anything.
Unfortunately everything I've applied this pattern to is proprietary, so I can't share any code examples.
It was mostly about putting the "imperative" part closer to main.go and putting business rules and things like that in the other files: Routing, serialization, validation, business rules, data transformation, command line argument parsing, configuration parsing, they all go in the "functional" part. Instantiating the HTTP server, reading argv, files and reading/writing to the database goes in the "imperative" part.
The major hurdle for me is frameworks and libraries that often want to be used in an imperative way. Some routers, for, instance could be purely functional, but they often want to instantiate the HTTP server themselves. Not a big issue in practice in Golang, though.
Also: the database. To keep purity in a simple way, you gotta wrap the imperative parts and use callback-ish structures. This part is often where I cheat. But with a proper abstraction it's great. I hate to say the forbidden word here, but a "monadic" abstraction you can have your cake and eat it. I never did it in Golang and I have no idea whether it would work, but, I did it in C# and it was a breeze.
I agree with your assessment about testing. In the end it was incredibly easy for me to get 100% coverage on the functional part.
return DB.Query(...).ThenCommand(res => ...).ThenHttpSuccess(res => ...);
A big issue is that this was in very small project. Promises are not exactly pretty, and the code is not the easiest to maintain if you don't know how they work, which is why I "cheated" on the Golang app. I think someone else smarter might be able to figure this problem out too, though :)
Doesn't that boil down to a hexagonal design, ports & adapters, that is also mentioned at the end of the Boundaries video?
With subscription/contexts you get to really pare down the whole imperative shell part. They allow you to keep using reproducible pure functions, but memoization allows you drastically minimize state management and to cache intermediary values, so you're not recomputing the whole world each time. When the state changes the whole system elegantly only recomputes the pieces that are strictly necessary. You don't need to do anything manually - it all just happens automatically
For some reason I don't see this being picked up in generally outside of GUI/Web space. But the input side of state is semi-solved as I see it
The output side still needs "management" or a "shell" of sorts
(for context, I've only really played with this through Clojure's cljfx GUI library)
So just do that in code signatures too.
I tend to agree with https://svitla.com/blog/functional-programming-in-typescript....:
> Enter the functional core, imperative shell approach where you can do as much work as possible in a functional way, and isolate impure interactions on a thin, outer layer.
> The most prominent example of this approach is ReactJS where:
> - All components can be seen as pure functions that take props, state, and context as an input and return a React element as an output.
> - All “outside world interactions” take place on the ReactDOM render/hydrate function.
Hahaha...great choice of words for an article on functional programming =)
- https://hw.leftium.com/#/item/18043058 (127 comments)
- https://hw.leftium.com/#/item/29613107
---
A FAQ is "Are there example of non trivial apps written in this manner?"
- CodeMirror/ProseMirror [1]
- React [2]
[1]: https://codemirror.net/docs/guide/#:~:text=Functional%20Core...).
[2]: https://svitla.com/blog/functional-programming-in-typescript....
The funtional core is mainly your "use cases"/"services": they the core of your software because they deal with your domain model. They are functional in that they do not keep state (state belongs to the DB).
The imperative shell is all about the "web controllers" and "db concrete implementations", and whatever i/o is needed. So, in your controller you read your POST request and retrieve the params you wanna work with; you also handle the rendering of templates if needed. Your db concrete implementation is all about SQL (or ORMs).
Functional programming is not too important in the core, a lot of benefits arise without it, e.g.: I can replay a prod session from the events generated and get the exact same behaviour. Makes debugging a breeze.
One advantage of having the state in one big place is that I can serialize it, send it to another thread that serves a UI from it.
I find it typically easier to ask “can I extract data and functions out of here” rather than “can I extract the side-effects out of here”. And I found that asking this question too early can be detrimental if you’re still trying to figure stuff out.
I don’t follow the paradigm strictly and I often write code in order to understand something. The first draft is typically just procedural with minimal abstractions. It’s more efficient that way.
Only after it’s working I start to pull out the pure bits and write tests for them.
> This is the cadence of the entire project so far. I tend to bloat up this outer file to something like 250 lines in some cases. Then I extract behavior out, moving it into unit-tested classes, figuring out how to make it more functional.
Skip to about 11:40 in the video. (There is some additional explanation right before that.)