How not to write a pipeline
cohost.org
cohost.org
What irritated me was the underlying sound that programmers of category 1 are always building bad solutions and programmer 2 has to fight to save the day. It feels like this person has worked to long in a bad position and started to see the world in this simple black and white form.
To be fair I’ve been there before myself. I maintain a ton of build pipelines and build tools. And for a while I had the world view of them against me. Them, who by design create bad build scripts which never scale and never survive a second use-case change and me, the unsung hero who fights the verbal battle why one should not write build output into the build source directory etc.
For me it boiled down to understand that these other people where not standing in line to make my life harder. They simply didn’t know or had no capacity to learn this because they also had their fair share of problems in their respective domains etc.
I think there is some good information in this post. But I would have liked if it had been wrapped differently.
I think a good general rule is: for a single sentence, not capitalizing the first letter (and not using a period) is fine. Longer than that and it both makes parsing more difficult and looks strongly like an affectation. There is a practical purpose for sentence start/end punctuation.
Fundamentally, no-one likes working with other people's code or solutions. They always think that they'd have done it better if they'd had the chance.
I do two things to dig out of this particular hole:
- Have some (relatively) objective standards by which I judge code/a project
- Have some empathy for people who were probably just doing their best, in the most multivariate sense ("I wrote this in Clojure so I didn't burn out", "I had to ship this in three days and I was just back from paternity leave", "I was onboarding a junior engineer and dumped this spaghetti in a spare 3 hours to mollify an important client"
#2 is very useful because it snaps me out of contempt and helps prevent me from spreading that culture to the engineers around me. But #1 is actionable in that when I do get time to "do things right" I have some values to aim at. Mostly they're:
- Does this code build/respect a coherent mental model of the problem space
- Is it possible to represent invalid state
- Are surprises/pitfalls noted (I'm not a big commenter in general, but this is a case where I find them useful)
- Does this code consider likely engineer workflows (e.g. is it easy to add another case to handle, is it easy to enable debugging, etc.)
Notably none of these are like, "methods are too long" or "variable names are too short" or whatever. I find orthodoxies like these just serve to create in/out groups and have no impact on code quality. Or, another way to say this is that if you're building a coherent mental model, your methods are probably the right size.
I dislike this generalization, as it doesn't go down to the gist: Solutions that adhere to "some" well-known patterns or ways to build things are more likely to be accepted by other fellow developers.
I, personally, can happily work with other people's code or solutions. :)
I think there's a frequent cause to this... It's difficult to know the requirements and constraints in place when that code or solution came to be. Many of us would have done something very similar, at least in spirit, under the same conditions. But those conditions constantly change and we tend to judge solutions based on current conditions. Try judging your own solutions by current conditions years after developing them and they sometimes won't look as good as you remember then being. When doing that, it's tempting to think "but x wasn't available and we didn't know y." With other people's solutions, we often lack the knowledge to be aware of those justifications. Investigating to find some of those justifications has helped me better figure out when replacing something is the right or wrong thing to do.
but yes, this is the lament of the senior developer or manager. the ideal is that people have the freedom to explore, and that they feel good for figuring things out, but that ultimately they're subtley shepharded such that they don't fall into infinite gravity wells or crash the ship along the way.
I say this as someone who spent 10 years dreaming of working for The Onion.
I feel this type of tone can work as a brief intro/aside, but is rarely suited for long form. I certainly don't think it befits a technical topic.
Despite the prose, the article is dripping with actual production experience, and many things I wish I had known earlier. To craft a good design, one must be aware of all the problems that the design should fix in order to be good (and accept that settling early into a design will prevent some problems from ever being fixed).
I would add that some queue solutions (such as Azure) implement a lease system where a dequeued message re-appears at the head of the queue after a while, unless its removal is explicitly confirmed.
it's a little confusing to me that the author seems to object to "pipelines" and then equate them with messaging-queues. for me at least, "pipeline" vs "workflow-engine" vs "scheduler" are all basically synonyms in this context. those things may or may not be implemented with a message-queue for persistence, but the persistence layer itself is usually below the level of abstraction that $current_problem is really concerned with. like the author says, eventually you have to track state/timestamps/logs, but you get that from the beginning if you start with a workflow engine.
i agree with author that message-queues should not be a knee-jerk response to most problems because the LoE for edge-cases/observability/monitoring is huge. (maybe reach for a queue only if you may actually overwhelm whatever the "scheduler" can handle.) but don't build the scheduler from scratch either.. use argowf, kubeflow, or a more opinionated framework like airflow, mlflow, databricks, aws lamda or step-functions. all/any of these should have config or api that's robust enough to express rate-limit/retry stuff. almost any of these choices has better observability out-of-the-box than you can easily get from a queue. but most importantly.. they provide idioms for handling failure that data-science folks and junior devs can work with. the right way to structure code is just much more clear and things like structuring messages/events, subclassing workers, repeating/retrying tasks, is just harder to mess up.
I've seen the "queue mess" architecture many times myself but it's been too embedded to easily migrate to something else.
But I find that queues are often misunderstood/misused. You have to be careful not to use them in systems where the producer cares about the output (true story - I've had someone earnestly suggest an "ack queue," essentially implementing TCP-over-SQS), you absolutely cannot use them in latency sensitive contexts (if you need an async job pool, use a stack. Your peak load customers will thank you), and you have to be careful about idempotency, ordering, and delivery guarantees.
Also to put on my own arrogant hat, most of the time I've seen linked queues it's been better to orchestrate the system as a state machine.
I’ve long, long since given up getting the, to realize that separate build plans are only accomplishing one thing: Enabling (as in codependency) a bunch of bad, slow tests in two sections of the code to persist instead of needing to be rewritten.
This project has no proper reason to be more than a dozen modules, and we’ve spent time on tooling to reduce the burden of managing so many repos at once.
If you don’t have a small mountain of builds you have to compose, you don’t have the queuing problems that go along with them.
Having read the original title and the first fourth of the article, I still don't know what it is about.
But we take gratuitous parens out of titles anyway and it looks like a mod did that later.
> you are
[...]
> employee number four and the ceo keeps writing code on weekends
I am also not sure if the author has valid assumptions about business requirements - thumbnails being not generated doesn't look like a reason to be waked up at night to me.
Also you usually have an intern or two in the support team who can re-run failed jobs and don't need to waste time of your devs or ops or devops for that.
Thumbnails not being generated might not be worth an early morning alarm, but running out of disk space might be, or not getting to do other work because it's blocked by the failure of thumbnail generation.
This means: merge the PR first, let it go live, use your working students or interns to rerun stuff, wait for a month - if it is still happening, then you have a proof of a problem that needs to be fixed.
Disk space: use your monitoring tool to proactively warn you when the free disk space is below of 20% or is reducing too quickly.
If some other work is blocked by failed thumbnails, this is a logical bug and not the consequence of a message bus. This stuff has been blocked even before the introduction of the message bus anyways.