What functional programmers get wrong about systems
iankduncan.com
iankduncan.com
Even if it is, or isn't ai, the content is in fact correct.
There is no 'one' state of your application, unless you literally do maintenance window deploys and 0 queues and keep everything sync.
> Message queues are version time capsules
> Event sourcing: the version problem as a way of life
> Temporal and bitemporal databases: time as a first-class citizen
> Semantic drift: the type didn’t change, but the meaning did
> Knowing what’s running changes everything
> What if the old code just kept working?
> The right tools, pointed at the wrong level
Presentation matters as much as content. Particularly if you want somebody to read 10,000 words, making that reading go down smoothly is a good thing to strive for. If this was by chance written by a human who happened to have absorbed LLM-like writing tendencies, I would still find fault in this article for how it is written, and would suggest they spend more time revising it rather than publishing a 5k-10k word technical article daily. Much like writing code, sheer lines written is not the goal; the actual goal is to succinctly and clearly represent your ideas in as refined a form as possible. This article dragged on and on and on, with fatiguing prose, for an idea that can be well expressed without such length.
If you disagree and find this to be good writing, you are entitled to your opinion, but nonetheless this is my own feedback on the article.
> I felt as though this article was, through its headers and overuse of specific rhetorical devices, constantly trying to grab my attention in that same shallow manner.
I think perhaps you're quick to assess a certain type of writing, which many see as done quite well and in a way that's approachable and is good at retaining interest, as AI. Perhaps you just don't like this type of writing that many do, and AI tries to emulate it, and you're keying on specific aspects of both the original and the emulation and because you don't appreciate either it's hard for you to discern between them? Or maybe there is no difference between the AI and non-AI articles that utilize these, and it's just your dislike of them which colors your view?
I, for one, found the article fairly approachable and easy to read given the somewhat niche content and that it was half survey of the current state of our ability to handle change in systems like these. Then again, I barely pay any attention to section titles. I couldn't even remember reading the ones you presented. Perhaps I've trained myself to see them just as section separators.
In any case, nothing in this stuck out as AI generated to me, and if it was, it was well enough done that I don't feel I wasted any time reading it.
Good technical writing allows you to get to and understand the point in a minimum of time, has a clear and obvious structure, and organizes concepts in such a way that their key relationships are readily apparent. In my opinion this article achieves none of these things (and it also is just bad insofar as its thesis is confused and misleading in a very basic way—namely the relationship between functional programming philosophy and distributed systems design is far more aligned than it suggests, and it sets up a false dichotomy of FP versus systems, when really the dichotomy is just one of different levels of design (one could write the exact same slop article about what OOP "gets wrong" about systems—it gets it "wrong" because low level programming paradigms techniques are in fact about structuring programs, not systems, and system design is largely up to designers—the thesis is basically "why don't these pragmatic program-leave techniques help me design systems at scale" or in other words "why don't all these hammering techniques help me design a house?")
I came away with some additional understanding of the problem, and thinking there are various nascent techniques to address this problem, none of them entirely sufficient, but that it's being worked on from multiple directions. I'm not sure the article was aiming for more than that.
Thanks for clarifying, in this case it might be comparing apples to oranges as I'd be surprised if most people approach they're writing like he does
So either his prompt is making the writing more palatable, or maybe he's just prolific.
In either case, I would count this as a w
My biggest problem with the article is the title, which is somewhat the inverse of clickbait--functional programmers are his apparent intended audience but the content is much broader.
> the author has written five engineering articles in the last week
considering you can only see when the articles are published, not written, I'd say you're trying too hard...
If it looks like a duck...
It isn't impossible that it's AI, but assuming writing and publishing happen at the same time would also lead you to conclude that Anne Frank wrote from the afterlife.
Plenty of people out there are happy to let AI do most of the work while claiming it as their own.
Makes me think - what if all of this was written by an LLM agent?
It's just one of the signals - not the primary one.
10k words for "migrating code and data is a headache". Yeah. Next.
Too many smartasses commenting on the skills (or lack thereof) of others just because of "AI".
Could be shorter though, and the title might not be 100% spot on and a bit clickbaity indeed.
Also, it does not shout "AI generated" to me. The style is pleasant enough (but for the repetitions).
I honestly can't relate this comment with that article. Have we read the same thing?
2. "What a spurt of productivity!" = a petty personal snide that's not in good faith at all.
3. You provide nothing in terms of what this article gets wrong or the technicalities of it. If you want to discuss flaws with the article, be specific instead of "this screams AI! clickbait rags!"
As someone who works with a LOT of AI-generated content, I'm pretty sure this is not AI-written.
But the solutions and tools functional programming provides help to have higher verifiability, fewer side effects and more compile-time checks of code in the deployment unit. That other challenges exists is fully correct. But without a functional approach, you have these challenges plus all others.
So while FP is not a one-size-fits-all solution to distributed systems challenges, it does help solve a subset of the challenges on a single system level.
These challenges can be solved by the usual tools of FP, but this requires each version of the system to be explicitly aware of the data schema varieties used by all earlier versions that are still in use. Then it's a matter of interpreting earlier-versioned data correctly, and translating data to earlier schema varieties whenever they may have to be interpreted by earlier versions of the code.
(It may also be helpful to introduce minimally revised releases of the earlier codes, that simply add some amount of forward compatibility re: dealing with later-released schemas, while broadly keeping all other behavior the same to avoid unwanted breakage. These approaches are not too hard to implement.)
> Here is the central claim: the unit of correctness in production is not the program. It is the set of deployments.
The thesis essentially boils down to: functional programing paradigm, type systems, strong interfaces, etc, are all fantastic tools for ensuring the correctness of a program, but the system is not a program, and so these tools are necessary but not sufficient to ensure the correctness of a distributed application.
And doing the things the article suggests would be mostly useless, because the problem is always the boundary between your neat little world and the outside. You can't solve the semantic drift problem by being more careful about which versions of the code you allow to coexist, because the vast majority of semantic drift problems happen when end users (or third parties you integrate with) change their mind about what one of the fields in their/your data model means. There are actually bitemporal database systems that will version the code as well as the data, that allow you to compute what you thought the value was last Thursday with the code you were running last Thursday ("bank python"). It solves some problems, but not as many as you might think.
even setting aside the memory safety stuff, having sum types and compiler-enforced error handling is also not nothing...
I still want to believe that future programming languages will be capable of tackling these issues, but exploring the design space will require being cognizant of those issues in the first place.
I don’t think the criticism of the author applies LISP-style functional programming, which is much more accommodating to embracing chaos.
In Haskell you can “embrace chaos” exactly as much as you desire. It’s a general purpose programming language.
It was intended as a criticism of the article, that its whole assumption “FP == making invalid states impossible to represent” is incorrect.
I recognize that it’s very much possible to embrace chaos with Haskell, and I should probably have worded that better.
Looks like the term "functional programming" has been watered down so much that now it is as useful as OOP: not at all.
Look, what matters is pure functional programming. It's about referential transparency, which means managing effects and reason about code in a similar way you can do with math. Static typing is very nice but orthogonal, ADT and making illegal states unrepresentable are good things, but all orthogonal.
Fundamentally - all your system after a certain scale becomes distributed systems, accuracy on a local level is usually not where systemic risks are coming from. A distributed system of fully accurate and correct local programs can implode or explode. Migration, compatibility and system stability are surprisingly dependent on engineer experience till this day.
This isn’t unique to Haskell or functional programming at all. I’ve seen programmers from JavaScript to C++ suffer the same fate: tied into the boundaries of a single process they try to verify the correctness of their systems informally. By writing documents and holding long meetings or convincing themselves by prototyping. It’s only a matter of time before the find the error in production.
Hopefully it’s of no real consequence. The error gets fixed and it’s a week or two of extra work. But some times it triggers an outage that causes SLO’s to overshoot and SLAs to trigger which cost the company money. That outage could have potentially been avoided.
But it’s tough getting developers to see the benefit in model checking or any kind of formal methods. The hubris is often too great. It’s not just functional programmers! Although it surprised me when I was working with Haskell developers that so many were resistant to the idea.
(If one has experience and understands functional programming, learning TLA should be rather straight forward)
In practice it’s a bit tricky to use because it doesn’t enforce encoding the schema identifier in the serialized payload. Oops.
But between that, automated typing from examples, and refinement it might be possible to develop a system that can manage to validate the serialized data against the parser and have the serialization data appear in the type. At that point you could query the whole system.
Even if you have a web application with only one server, and an embedded database, and no other components, you still have a distributed system. The browser is part of that system.
But none of these things are functional programming? This is more the tradition of 'expressive static types' than it is of FP.
What about Lisp, Racket, Scheme, Clojure, Erlang / Elixir...
> The immutability of the log is the entire value proposition.
In practice, much of the article seems to be about the problems introduced by rigid typing. Your quote, for instance, is used in the context of reading old logs using a typed schema if that schema changes. But that's a non-issue in the FP languages mentioned above since they tend towards the use of unstructured data (maps, lists) and lambdas. Conversely, reading state from old, schema-incompatible logs might be an issue in something like Java or C++, which certainly are not FP languages as the term is usually understood.
So overall, not really an FP issue at all, and yet the article is called "what functional programmers get wrong". The author's points might be very valid for his version of FP, but his version of FP seems to be 'FP in the ML tradition, with types so rigid you might want to consult a doctor after four hours'.
Haskell doesn’t have this problem. None of the “rigid typing” languages have this problem.
You are in complete control of how strictly or how leniently you parse values. If you want to derive a parser for a domain-specific type, you can. If you want to write one, you can. If you want to parse values into more generic types, you can.
This is one of those fundamental misunderstandings that too many programmers have, and it seems like it’ll never die.
You don't pick up Haskell just to spend all your time passing around unstructured data, any more than you opt into the overhead of TS just so you can declare everything as `any`.
You do not understand how typing works in Haskell. You are free to work with primitives as much as you like.
> You do not understand how typing works in Haskell.
Possible! But a little hyperbolic, perhaps. I think we're more likely to be talking past one another.
> You are free to work with primitives as much as you like.
Sure, but what's the point? If all of your functions are annotated as `myFun :: a -> b`, where a and b are arbitrary type variables, why are you writing Haskell? You're effectively writing Ruby with extra steps. You're nether getting the benefits of your rigidly typed language, nor the convenience of a language designed around dynamism.
Yes, ML-esque type systems are quite neat and flexible. But the more granular the typing you opt into, the more of the usual cost of typing you incur. Typing has inherent cost (and benefit!). And if you're not into the typing, ML-esque typed languages are a curious tool choice.
So to return to the original point, if you're passing data around in Haskell, you have more likely than not opted into some level of typing for that data - else why use Haskell - and will run into the exact issues with rigid type systems mentioned above. Can you parse and type cast and convert and what not? Sure. But no-one ever said that you couldn't, and that's precisely the busywork that dynamic languages are generally designed to lead you away from.
Direct, certainly. More than would be typically expected socially. But I don't think it's hyperbolic — I think there is genuine fundamental misunderstanding here.
The best source which I believe dispels your misunderstanding is this one: https://lexi-lambda.github.io/blog/2020/01/19/no-dynamic-typ...
If I may be so bold, I'd posit the misunderstanding is on your part. No one is saying things are impossible to model in rigidly typed systems - this is your key misapprehension about what is being said. What I'm saying is that different languages have different paths of desire, and the kinds of problems identified in the original article are more the kind of problems that tend to crop up with heavy use of types, than they are the kind of problem that has much of anything to do with functional programming.
You're thinking categorically, but I am not, so we're talking at cross-purposes. Perhaps too much static typing has crept into your thinking! (I jest, of course! :) )
> What I'm saying is that different languages have different paths of desire, and the kinds of problems identified in the original article are more the kind of problems that tend to crop up with heavy use of types, than they are the kind of problem that has much of anything to do with functional programming.
I don't think this is correct at all. I don't think TFA has anything at all to do with types or FP (despite the clickbaity title), as numerous other people here have already pointed out. The article isn't attacking rigid types. The author's point is that no single-program analysis — typed or untyped — covers the version boundary (or system boundaries more generally).
A Haskell service that receives an unknown enum variant doesn't have to crash — you parse the cases you care about and ignore the rest. The "path of desire" you're describing isn't a property of the language.
I suppose "path of desire" here is a matter of opinion. In my experience, crashing on unknown inputs is not idiomatic Haskell, nor is it desirable.
Why would understanding his argument necessarily mean finding it persuasive or exhaustive?
> you parse the cases you care about and ignore the rest
What language would not allow this?
> The "path of desire" you're describing isn't a property of the language.
If a tool tends, more often than not, to lead to certain use, is that not a property of that tool? It is theoretically possible to use a hammer for interpretive dance, sure, but it doesn't seem to happen nearly as often as banging the hammer on things.
Equally, I think it's pretty easy to see how a language designed for robust typing is going to lead to, more often than not, robust typing.
I rather think you're engaging with a point that no one ever made - that typed languages are inherently incapable of dealing with uncertain, incomplete, or variable data - in lieu of the argument that was actually made - that languages with rich DX around rigid typing encourage an architecture that's rigidly typed, and that rigidly typed codebases tend to come up against predictable issues.
The original article identities a series of such issues and misattributes them to FP, when they don't have much to do with FP at all. That's all I was saying.
Alexis King is a woman.
> that languages with rich DX around rigid typing encourage an architecture that's rigidly typed, and that rigidly typed codebases tend to come up against predictable issues.
I agree with neither of these points.
Fair pick up! I never read bylines, so I ought to stick to gender neutral pronouns.
> I agree with neither of these points.
Perfectly reasonable. I'm not going to imply you're ignorant for disagreeing.
The application that assumes that key "foo" is in the map, and crashes if it's not, is just as brittle[1] as the one that assumes that the data deserializes into a struct with the field foo present.
[1]: In practice, it's more brittle because the crashes are more unexpected and at runtime. Javascript has a legacy of this...
And the program that assumes that foo is a string when it's an integer yadda yadda yadda.
What's better, that code containing erroneous baseline assumptions about the data it's handling stays live, or that it safely aborts before it corrupts any of that data? All software must deal with errors at some point or another.
Take a web endpoint. If it is expecting a foo, and it instead gets bar, what is it intelligently supposed to do with that bar? We're squarely in the realm of a fatal error (to that request), the only question is how much boilerplate is going to be required to abort.
In most languages, the answer is endless validation and error handling at boundaries. In Erlang, on the other hand, such a function would simply crash. All Erlang code runs in green threads, usually overseen by supervisor trees. The supervisor tree organises the quick return of an appropriate error to the request, while it safely restarts the worker thread.
This kind of architecture requires no boilerplate or external libraries in Erlang or Elixir. It's built in. The VM is optimised for remaining robust under load even while orchestrating millions of workers. Immutability prevents any data races between all these threads, or any need for locks at all, ever. The end result is that all business logic remains short and focussed on the happy path. The DX is out of this world.
Read up on the BEAM sometime, you might find it interesting. There's much more to FP than endless typing.
> If you have a web application with more than one server, you have a distributed system. If you have background job workers, you have a distributed system. If you have a cron job, you have a distributed system. If you talk to Stripe, or send emails through SendGrid, or enqueue something in Redis, or write to a Postgres replica, then you are (I regret to inform you) operating a distributed system.
As soon as you have at least one client talking to your server you have a distributed system.
It makes me a little sad that Erlang/Elixir is kind of the only platform that does this though...
It's not. These are called green threads and under the hood it's not really threads. It's literally the same thing as async await in python and nodejs.
It's just different perspectives on the same concept of concurrency. Go and Erlang implicitly have calls to concurrency, while nodejs it's implicit.
Erlang processes, goroutines, and async tasks are all cooperatively scheduled user-space execution units multiplexed over OS threads. That is the same underlying concurrency primitive. There is no new physics here.
Erlang’s real contribution is that it forbids shared memory, bakes message passing, supervision, and restart semantics into the runtime, and forces a particular design discipline. That’s a design rule, not a different execution model.
You can approximate Erlang’s semantics on top of async/await or goroutines; you cannot approximate preemptive shared-memory threads on top of Erlang. That asymmetry tells you where the real difference lies.
Treating policy and guarantees as if they were a different kind of concurrency entirely is what causes people to mystify Erlang instead of understanding why its constraints work.
Also, it’s not the same thing as async nodejs. Node.js is a fully cooperative system, something CPU intensive can hog the thread and starve the other tasks. Erlang processes are preemptive; every process gets N reductions and then it is parked and the next process can make progress. They’re closer to OS processes than goroutines.
Was the central thesis supposed to be that it's impossible for any tools to ever represent complex migrating systems with well-defined data? Because that's not true. Or that it's not practical or cost-effective to do so? Or was there a central point?
First put all your services in a monorepo have it all build as one under CI. That’s a static check across the entire system. When you do a polyrepo you are opening up a mathematical hole where two repos can be mismatched.
Second don’t use databases. Databases don’t type check and aren’t compatible with your functional code written on servers.
The first one is stupid… almost everyone just makes this mistake and most of the time unless you were there in the beginning to enforce monorepo you have to live with someone deploying a breaking change that effects a repo somehwere else. I’ve been in places where they simply didn’t understand the hole with polyrepos and they went with it despite me diagramming the issue on a white board (it’s that pervasive). Polyrepos exist for people who like organizing things with names and don’t understand static safety.
And it gets worse. So many people don’t understand that this problem only gets worse over time. The more repos you have in your polyrepo the more brittle everything becomes and they don’t even understand why.
The second one is also stupid but you had to be there in the very very beginning to have stopped it. The inception of the concept of databases should have been created such that any query you run on it needs to be compiled to run on the database and is thus statically checked. This is something that can easily be done but wasn’t done and now the entire World Wide Web just has to make do with testing. Hello? Type checking in application code but not in query code? Why?
That’s like half the problems with distributed systems completely solved. And these problems just exist because of raw stupidity. Stuff like race conditions and dead locks are solvable too but much harder. But these issues are obvious. What’s not obvious are the aforementioned problems that almost nobody talks about even though they are completely solvable.
This is a good point, I've never really thought about this carefully before, but that makes so much sense.
That isn't very useful by itself. What's your suggested alternative that aligns with your advice of "don't"? How does it deal with destructive changes to data (e.g. a table drop)?
>How does it deal with destructive changes to data (e.g. a table drop)?
How does type checking deal with this? What? I'm not talking about this. I'm talking something as simple as a typo in your sql query can bring your system down without testing or a giant orm that's synced with your database.
I'm not saying distributed systems are completely solved. I'm saying a huge portion of the problems exist because of preventable flaws. Why talk about the things that can't really be easily solved and why don't we talk about the things that can be solved?
Even if you were to reduce the database to a simple API, the question then remains how do you make sure to version it along with the other portions of the system that utilize it to prevent problems. The point of the article seems to be to point out that while this is a much harder problem (which I think you are categorizing as "things that can't really be easily solved"), there are actually solutions being developed in different areas that can be utilized, and it surveys many of them.
Right. But we haven't even have square one solved which is the easy stuff. That's my point.
>Even if you were to reduce the database to a simple API, the question then remains how do you make sure to version it along with the other portions of the system that utilize it to prevent problems.
I said monorepo and monodeploys in this thread. But you need to actually take it further then this. Have your monorepo be written in a MONOLANGUAGE, no application language + sql, just one language to rule them all. boom. Then the static check is pervasive. That's a huge section of preventable mistakes that no longer exist, now that type that represents your table can never ever be off sync.
I know it's not "practical" but that's not my point. My point is that there's a huge portion of problems with "systems" that are literally obviously solvable and with obvious solutions it's just I'm too lazy to write a production grade database from scratch, sorry guys.
And that helps when you are dealing with schema changes that need to be rolled out at AWS, your local DB, a Kafka cluster, how? The whole point of this article was how to approach the problem when there are different components in the system which make a monorepo and what it provides for this infeasible or impossible.
> I know it's not "practical" but that's not my point. My point is that there's a huge portion of problems with "systems" that are literally obviously solvable and with obvious solutions it's just I'm too lazy to write a production grade database from scratch, sorry guys.
The article talks about database solutions that help with this problem.
I'm uncertain how to interpret your responses in light of the article, when they seem to be ignoring most of what the article is about, which is solving exactly these problems you are talking about. Is your position that we shouldn't look for solutions to the harder problems because some people aren't even using the solutions to the easy problems?
That is not what I’m arguing against.
My point is more fundamental: we deliberately designed away static safety at the foundation, and then act surprised that “systems problems” exist.
Before Kafka versioning, schema migration strategies, backward compatibility patterns, or temporal reasoning even enter the picture, we already punched a hole:
Polyrepos break global static checking by construction.
Databases are untyped relative to application code
SQL is strings, not programs
Deployments are allowed to diverge by default
That entire class of failure is optional, not inherent.
When I say “we haven’t solved square one,” I’m saying: we skipped enforcing whole-system invariants, then rebranded the fallout as unavoidable distributed systems complexity.
So when you say “the article already offers solutions,” you’re misunderstanding what kind of solutions those are. They are mitigations for a world that already gave up on static guarantees, not solutions to the root design mistake.
I’m not claiming my position is practical to retrofit today. I’m claiming a huge portion of what we now call “hard systems problems” only exist because we normalized avoidable architectural holes decades ago.
You’re discussing how to live in the house after the foundation cracked.
I’m pointing out the crack was optional and we poured the concrete anyway.
I’m telling you this now so you are no longer uncertain and utterly clear about what I am saying and what my position is. If you are unclear please logically point out what isn’t clear because this phrase: “ The article talks about database solutions that help with this problem.” shows you missed the point. I am not talking about solutions that help with the problem, I am talking about solutions that make a lot of these problems non-existent within reality as we know it.
I ask you what to use instead, and how to deal with datastore versioning.
You say you're talking about how we don't have type safety that extends to the remote systems we're interacting with.
I ask how that helps versioning problems with these systems where you need to deal with applying changes across distributes systems, which specifically is not solved by having types in lockstep definition between systems, because in application of change there are problems to work thought.
You note we did all this deliberately and we didn't have to. I keep asking you what the other option is, because you keep acting like there is one, but refusing to give an example of what that would be, because a monorepo is no solution for the problems being discussed here in the article, which to be clear, are not limited to code.
You've made it very clear you think we should have done "something" else, but refuse to articulate what that is. If it's not known, or not explored, then I posit we didn't "choose" this path, it's the path that was open to us.
> You’re discussing how to live in the house after the foundation cracked.
You keep saying we should have used something else for the foundation that wouldn't crack, but refuse to explain what this mythical material is.
What is your proposed alternative, or are you just waxing theoretical?
I am not saying the problem is impossible. I am saying the problem is obvious, stupid, and solvable in theory, and that the only actual solution is to rewrite the foundation from scratch. That is precisely why there is no practical path forward. The impracticality is the point.
When you keep asking me to name an alternative, you’re implicitly assuming I’m advocating for some incremental migration or deployable fix inside the current ecosystem. I am not. There isn’t one. If you want whole system static guarantees, you need a database that is designed from the beginning to be part of the same language and type system as the application. Queries are programs. Schemas are types. The datastore is a compiled artifact, not a remote string interpreter. That requires a fundamentally different database.
That is the alternative. And it is completely unrealistic to retrofit into the existing world. Which is why we are stuck.
So when you say I’m “waxing theoretical,” you’ve missed the entire point. The theory is the indictment. The fact that the solution is obvious but unusable is exactly the problem. We built ourselves into a corner decades ago, and everything we now call “systems engineering” is about managing the consequences of that decision.
The article is fine. It surveys techniques for surviving in the world as it exists. I am not disputing their usefulness. I am saying those techniques exist because we accepted a broken foundation and normalized it. That distinction matters.
You keep arguing as if I’m refusing to answer your question. I’m answering it directly. The answer just isn’t one you like. The only real fix is a ground up rewrite of the data layer and its relationship to code, and that’s never going to happen at scale. That’s the conclusion. Understand?
If you think that means the path we’re on was the only one available, then we fundamentally disagree about what “choice” means in system design.
You’re still trying to refute a design critique by demanding an escape hatch, which only proves you never understood what was being critiqued.
Yeah, those pesky laws of physics, getting in the way of purity.
You simply cannot deploy simultaneously to an active fleet of servers.
Clients that our outside of your window of control obviously can't be part of the zone of purity. In that case the management practices in the article applies. You need to differentiate and control what you can, which is exactly what haskell does.
What I'm referring to is that a HUGE number of systems that can be in the zone of purity, aren't. Microservices architecture does not colloquially exclusivey refer to systems outside of our control, it refers to building a constellation of services WITHIN our control (think of a monolith within our control, but just broken down to microservices for the unintended purpose of introducing all these extra impure issues).
So a huge number of problems people face in distributed systems will be EXACTLY similar to the same problem faced by the phone with a client app simply because of bad choices and NOT because the component lies outside of their control. Like why am I facing this update problem between two services I control? Why should I even face this problem in the first place and why did I make this problem exist when I CONTROL both services? There's no excuse here.
That helps but is insufficient, since the set of concurrently deployed artifact versions can be different than any set of artifact versions seen by CI -- most obviously when two versions of the same artifact are actively deployed at the same time. It also appears to rule out the possibility of ever integrating with other systems (e.g., now you need to build your own Stripe-equivalent in a subdir of your monorepo).
> Second don’t use databases.
So you want to reimplement your own PostgreSQL-equivalent in another monorepo subdir too? I don't understand how opting not to use modern RDBMSes is practical. IIUC you're proposing implementing a DB using compiled queries that use the same types as the consuming application -- I can see the type-safety benefits, but this immediately restricts consumers to the same language or environment (e.g., JVM) that the DB was implemented in.
Simple, although I only mentioned repos should be mono, I should've also said deployment should be mono as well. I thought that was a given.
>So you want to reimplement your own PostgreSQL-equivalent in another monorepo subdir too?
I'm too lazy to do this but in general I want an artifact that is built. All sql queries need to be compiled and built and the database runs that artifact. And of course all of this is part of a monorepo that is monodeployed.
>I don't understand how opting not to use modern RDBMSes is practical.
It's not practical. My point is there's some really stupid obvious flaws we live with because it's the only practical solution.
Deploying your service graph as one atomic unit is not a given, and not necessarily even the best idea - you need to be able to roll back an individual service unless you have very small changes between versions, which means that even if they were rolled out atomically, you still run the risk of mixed versions sets.
It's not a given because you didn't make it a given.
>and not necessarily even the best idea - you need to be able to roll back an individual service unless you have very small changes between versions, which means that even if they were rolled out atomically, you still run the risk of mixed versions sets.
It is the best idea. This should be the standard. And nothing prevents you from rolling back an individual service. You can still do that. And you can still do individual deploys too. But these are just for patch ups.
When you roll back an individual service your entire system is no longer in a valid state. It's in an interim state of repair. You need to fix your changes in the monorepo and monodeploy again. A successful monodeploy ensures that the finished deploy is devoid of a common class of errors.
Monodeploy should be the gold standard, and individual deploys and roll backs are reserved for emergencies.
There are a ton of reasons it's not the best idea. This flies in the face of a lot of _better_ ideas.
Keeping changesets small so that it's easier to debug when something goes wrong? Blown out of the water by deploying everything at once.
Bringing every service up at once is a great way to create the coldest version of your entire product.
Requiring a monodeployment turns canarying or A/B testing entire classes of changes into a blocking rollout where any other feature work has to move at the pace of the slowest change.
> When you roll back an individual service your entire system is no longer in a valid state. It's in an interim state of repair.
The gold standard is that each version of your service can work with each other version of your service, because in The Real World your service will spend time in those states.
> Monodeploy should be the gold standard, and individual deploys and roll backs are reserved for emergencies.
No, because if it's still possible to mix versions in your services, then a monodeploy doesn't actually solve any issues.
I actually am a big fan of fewer services and deploying bigger artifacts, but if you have multiple services, you have to act like you have multiple services.
The size of your monodeploy is orthoganol to the concept of monodeploy. You can make a large change or a small change.
In fact your deploy can be smart. For a specific service in a full system monodeploy when upgrading from v2 to v3 it can do some sort of diff on the source of a specific service and if there's no difference it goes from v2 -> v3 without a new build and uses the same artifact from v2 to v3. The entire point is though that this service (or the entire system) still goes from v2 to v3 and it tagged this way. This is an optional optimization for speed.
In fact, your compiler when building artifacts ALREADY does this. It caches huge parts of the build and reuses it. A deploy can do the same.
This is the important concept of a monodeploy: The static check; The integration testing. The verification of the ENTIRE system as a whole. Your monodeploy determines what new artifacts need to be recreated, what artifacts need to be reused... verifies everything, and deploys.
>Requiring a monodeployment turns canarying or A/B testing entire classes of changes into a blocking rollout where any other feature work has to move at the pace of the slowest change.
Again orthoganol. Your complaining that a monodeploy is slow. Integration testing and unit testing are also slow and take time. The monodeploy is for safety. If you're saying speed > safety, here's an idea: throw all testing out the window as well. That's a big speed up right there.
If your monodeploy is slow, work on speeding it up. Work on it being smarter and faster. Do you throw testing out the window because it's slow or do you work on speeding it up? Make the smart choice.
>The gold standard is that each version of your service can work with each other version of your service, because in The Real World your service will spend time in those states.
And that gold standard is stupid. We can do better. We can go to a state where different versions between different services don't exist. Only one monoversion. You throw that concept of different versions out the window then you also throw the possibility of a mismatch out the window as well.
You're trying to deal with an error. I'm saying make the error not exist.
>No, because if it's still possible to mix versions in your services, then a monodeploy doesn't actually solve any issues.
It's not possible to mix versions in a monodeploy because the whole concept of it is to have ONE version of everything. Let me be clear I'm talking about a MONOREPO + MONOBUILD + MONODEPLOY. If there's only one version of everything and it's all deployed than issues are solved under this model. At this point I think you just don't like being wrong.
>I actually am a big fan of fewer services and deploying bigger artifacts, but if you have multiple services, you have to act like you have multiple services.
A monodeploy doesn't preclude multiple services. You can still act like it's different services. A monodeploy + monorepo just makes sure there's ONE version of the entire system.
You're solution here is just saying you want to be able to deploy different versions of different services in a staggered way. You want different repos so different modules of the system can move out of step with everything else. Service A is at v23, Service B is at v32.
The only way to deal with this mismatch is to have complicated "versioning" system on top of that where API contracts between services only accept "backward compatible" changes. This works but it's also extra complication and extra restriction. You can no longer radically change an API because it can break a number of systems in different repos. You're stuck. Or if you're willing to deal with the fallout you can make breaking changes and accept the risk whilst under my system the risk doesn't even exist.
You are advocating for an idea that's definitively worse. But you'll never admit it, not right now anyway because basically you've dug your heals into the ground. At this point I've never seen a human who is so unbiased they are capable of proper reasoning to flip their stance. If we continue talking, you will continue to build logical scaffolding to support YOUR point rather then to support A point and it's pointless (punintended) to keep going.
I'm ok to keep going, but I think it's completely obvious to any neutral arbiter that the conversation is over and that your perspective is rationally worse.
There are definitely benefits to this approach. My coworkers do fall into the trap of assuming that all the services will be deployed simultaneously, which is not something you can really guarantee (especially if you have to roll one of them back at some point), so the monorepo approach gives them confidence in some breaking changes that it shouldn't (like adding a new required field).
I'm talking about it in the same sense as the "benefit" of using typescript over javascript. Not just a "benefit" but it's the obvious path, the obvious better way.
Everything about monorepos and polyrepos are basically mostly just debates about opinions and styles and preferences. But most people don't understand... the monorepo is definitively better. Don't think of it as a benefit, this makes that style of error literally impossible to occur.
>You still have to understand that services aren't deploying as an atomic unit and make sure that your network calls are forward and backward compatible.
The time between inception of a deploy and the termination of a successful deploy isn't solved. But a monodeploy solves an entire class of errors outside the boundary of an atomic deploy. Think about what's in that boundary and what's outside of that boundary? How long does a deploy take? An hour? How long are you not deploying?
That's they key, static checking can't fix everything and a monodeploy isn't a full guarantee of safety, but it does guarantee the impossibility of a huge class of errors in the interim time between successful monodeploys.
I guess my point is, monorepo vs. polyrepo... monorepo is the winner because it solves more classes of errors.
Ideas that helped me situate many of the issues we struggled with at various companies:
- distinguishing programs (the code as designed) from (distributed) systems (the whole mess, including maintenance and migration, as deployed)
- versioning code (reversibly) and data (irreversibly)
- walk-through of software solutions for distributed systems, and their blind spots wrt data versioning; temporal databases
- semantic drift: schema-less change
- Limitations of type systems for managing invariants
- co-evolving code and data; running old code on old data
The real benefit of systems thinking is recognizing whether an issue is unsolvable and/or manageable (with the current approach, or warrants a technology upgrade or directional shift). It's like O(n) evaluation of algorithms, not for space/time but for issue tractability.
There are a lot of reasons people don't succeed in systems thinking or in writing good distributed architectures, but functional programming is not one of them. If anything it's an inability to think functionally and to leak state and establish brittle dependencies absolutely everywhere that scuppers systems. Yes, FP has ton's of techniques that aid in good design at lower levels too, that doesn't mean it hampers you at high levels. In fact, much of what this article puts forward is a false dichotomy. None of the lower level FP principles and techniques like strong and expressive types systems are in conflict with the much more general principles of system design covered in the article (and they actually help you realize these principles—think version skew is bad in Haskell? good luck figuring out wtf is even happening in a tangled javascript async promises monstrosity undergoing version skew)
Also, slop, gross. At least edit some of that empty AI verbiage out of the final essay. It has the character and charm of a wet rag and adds zero information.
The core thesis sticks with me: existing tooling that helps programmers think about and enforce correctness (linters, compilers, type systems, test suites) operate primarily (and often exclusively) on one snapshot of a project, as it exists at a particular point in time. This makes those tools, as powerful as they can be, unable to help us think about and enforce correctness across horizons that are not visible from the standpoint of a single project at a single point in time (systems distributed across time, across network, across version history).
I feel like the issue of semantic drift is a bigger topic that probably has roots beyond systems architecture/computer engineering.
Also I found the writing style very clear. It helpfully repeats the core ideas, but (IMHO, this is subjective) doesn't do so ad nauseum.
I'm interested in reading other writing from the same author.
> This makes those tools, as powerful as they can be, unable to help us think about and enforce correctness across horizons that are not visible from the standpoint of a single project at a single point in time (systems distributed across time, across network, across version history).
I don't think there's something preventing you from constructing guardrails with type system & tests enforcing correctness that handles versioning.
I'm not buying the "unable to help us to think about" part. I argue that Rust's `Option`/`Result`/sum-type + exhaustive match/switch very valuable for proper error handling. You could define your own error type and exhaustively handle each case gracefully, with the bonus that now your compiler is also able to check it.
> “given the actual set of deployments that exist right now, is it safe to add this one?”
This problem has been right in front of our faces for decades, but I haven't ever seen it laid out like this, or fully realised it myself. Which is surprising, given the obvious utility of types for program-level correctness.
I think the next big leap forward in software reliability will be the development/integration of both the tooling to accomplish this, and the cultural norms to use them. Automating this (as far as possible -- they do note that semantic drift is immune to automatic detection) will be as big a deal as the concept of version control for code, or CI/CD.
There's lots here that I need to dig into further.
But if the type system need to cover all these components, then they start coupling again.
Coupling is not necessarily a bad thing as long as it gives good developer UX. If the database is tightly coupled with programming language, then it looks like ORM but better. And it probably can also reduce CRUD biolerplate or N+1 issue etc.
Related, there is SpacetimeDB that make backend run within database, and the backend code is highly coupled with SpacetimeDB's own API
For db stuff - what if we flipped the way we write schemas around. A schema is its something that you derive at a current state, rather than starting by writing a schema and then generating migrations? So you can reason over all of it rather than a particular snapshot?
[0] https://docs.datomic.com/datomic-overview.html#information-m...
Gameplay tags for example [0], and quite a lot of SQL.
[0] https://dev.epicgames.com/documentation/en-us/unreal-engine/...
The value of a schema in a db like Postgres isn’t in the description of the storage format or indexing, but the invariants the database can enforce on the data coming in. That includes everything from uniqueness and foreign key constraints to complex functions that can pull in dozens of tables to decide whether a row is valid or should be rejected. How do you derive declarative rules from a turing complete language built into the database?
Eg some table Users -> you start with `add user_id` , `add org_id`, `remove org_id` for example, so then the current state is `Users{ user_id }`. But you trust the compiler to derive that, and then when you want to do something with Users you have to scope into it, or tell it how to handle different steps in that chain.
Im not saying this is not equivalent at the end of the day, just if anything surfaces it this way, or makes it more ergonomic.
So when you call writeXY your caller has absolutely no need to know what actually happens. Catching and modifying old versions is just another morphism. You can even keep the layout and just accept version and payload as input.
You define an ecto schema and write your own migrations, but like the state of the app seems tied to that kinda snapshot of the schema. Its not like you have some lens into the overall chain of all the migrations together.
The correct answer here is that a database ought to be modeled as a data type. Sql treats data type changes separate from the value transform. To say this is retarded is an understatement.
The actual answer is that the schema update is a typed function from schema1 to schema2. The type / schema of the db is carried in the types of the function. But the actual data is moved via the function computation.
Keeping multiple databases around is honestly a potential good use of homotopy type theory extended with support for partial / one-way equivalences
Thank you to the author.
> A deploy pipeline that queries your orchestrator for running image tags, checks your migration history against the schema registry, diffs your GraphQL schema against collected client operations, and runs Buf’s compatibility checks: this is buildable today, with off-the-shelf parts.
that was largely successful at eliminating interservice compatibility errors, but it felt like we were scrambling around in a dark and dusty corner of software engineering that not many people really cared about. Good to see that there's a body of academic work that I was not entirely aware about that is looking into this.
The article successfully identifies that, yes, there are ways, with sufficient effort, to statically verify that a schema change is (mostly) safe or unsafe before deployment. But even with that determination, a lot of IDLs make it still very hard to evolve types! The compatibility checker will successfully identify that strengthening a type from `optional` to `required` isn't backwards compatible: _now what_. There isn't a safe pathway for schema evolution, and you need to reach for something like Cambria (https://www.inkandswitch.com/cambria/, cited in the article) to handle the evolution.
I've only seen one IDL try to do this properly by baking evolution semantics directly into its type model, typical (https://github.com/stepchowfun/typical) with it's `asymmetric` type label: it makes a field required in the constructor but not the deserialiser. I would like to see these "asymmetric" types find their ways into IDLs like Protocol Buffers such that their compatibility checkers, like Buf's `buf breaking` (https://buf.build/docs/reference/cli/buf/breaking/), can formally reason about them. I feel like there is a rich design space of asymmetric types that are poorly described (enum values that can only be compared against but never constructed, paired validations, fallback values sent over the wire, etc.)
Another one that I think makes a pretty good attempt is the config language CUE and its type system (https://cuelang.org/docs/concept/the-logic-of-cue/), which allows you to do first-class version compatibility modelling. But I have yet to investigate whether CUE is fit for purpose for modelling compatibility over the wire / between service deploys.
But yes, interfacing between formally proven code and the wild wild West is always an issue. This is similar to when a person from a country with a strong legal system suddenly find themselves subject to the rule of the jungle.
The better question to ask is why did we choose to build a chaotic jungle and how do we get out of it
Im not sure what the point of the post is. Minimizing side effects in your code (keeping functions pure) is what gives you the flexibility he says FP misses?!
There are experimental systems like Unison that keep old versions of your code alive as data, which are fascinating solutions to the underlying problem.
The formula for this article was: truism about X followed by unrelated stuff about Y that the reader is supposed to come away thinking X is incompatible with.
There's certainly no personal anecdotes of difficult problems which would go a long way to show the author actually knows their stuff.
I’m beginning to really appreciate short articles with a few bullet point takeaways.
With respect to version control across systems, when you get into serious stuff where mistakes are measured in lives and/or three commas, there’s just a lot more simplicity in the design of those systems than most people think.
Really big systems often have very simple design principles at their core which are echoed through out the topology.
In secure code like the stuff signal uses, having a hash of the code that is attested by a network of servers and chained back to self-signed identities on the client is the only way to go.
Here’s the hash of the code that’s supposed to be running on my server, here’s the proof that I verified it with all of the hardware and software tools at my disposal from the server, and here’s that hash and attestation embedded into the app on your phone or laptop that’s connecting to my box.
If there’s an easier way to get some semblance of “my device is talking to the right code on the right box,” please enlighten me.
An FP programmer admitting they live in ivory towers. I never thought I'd see the day.
I love it. HN needs 10 times more of this.
This person has grown up - the rest of you? Get a grip.