Automating processes with software is hard
hardcoresoftware.learningbyshipping.com
hardcoresoftware.learningbyshipping.com
Exceptions or edge cases add additional states.
To fight the explosion of state count (and the intermediate states those generate), you have a couple powerful tools:
1. Identifying and routing out divergent items (aka ensuring items get more similar as they progress through automation)
2. Reunifying divergent paths, instead of building branches
If you want to go back and automate a class of work that's being routed out, write a new automation flow explicitly targeting it. Don't try and kludge into into some giant spaghetti monolith that can handle everything.
PS: This also has the side effect of simplifying and concluding discussions about "What should we do in this circumstance?" with other stakeholders. Which for more complex multi-type cases can be never-ending.
PPS: And for god's sake, never target automation of 100% of incoming workload. Ever. Iteratively approach it, but accept reaching it may be impossible.
(I'm a founder of an automation SaaS where we've made "human interface" one of the core features of the product)
The biggest part of automation in my experience is boiling down the inputs to a 'unique' state that automation can then use as inputs and be run on.
For computation to do work, it requires a property of consistency, and computers can only operate accurately and do work when properties of determinism are met. Also as the OP mentions, state can explode leading to sphagetti, this is why he mentions a sieve like approach based on similarity.
Some problem spaces can be fundamentally inconsistent, such as with some approximations (common methods used for such), which falls back to what amounts to guesses, heuristics, and checks in terms of exception handling. There are problem scopes that cannot be characterized too so no amount of exception handling will resolve the entire scope, which is why you need fallbacks in a resilient design.
If inputs cannot be controlled and uniquely differentiated, the automation fails in brittle ways, especially with regards to external change.
The main interface (with regards to your core features) would be language, or communication. There exists words right now that can have contradictory, and different meanings, where the same word may mean the opposite depending on context, and this is not a general consensus but an individual one (where the individual may be misusing it).
That breaks the 1:1 mapping required for determinism, and AI weights mimicking neurons have a narrow approximation where it may work under a narrow set of circumstances but no computer today can differentiate when the inputs are the same but have two or more, different states mixed in (many people forget that absence of a state is a state too) and then decompose them. Abstract decomposition seems to be something only humans are good at, and I'm glad this is the case otherwise none of us would have jobs.
Do you have any examples of this? Like an example project?
> never target automation of 100% of incoming workload. Ever.
A new product owner came in last year and wanted to "automate everything". They wanted a web page where builds, branches, commits, and more were all on one page that showed exactly which commit was deployed where, by whom, etc etc. They wanted this extravaganza for a 2-person application that was in maintenance with no new features.
They're also the kind of person who consults chatgpt or copilot on a subject while you're explaining that subject to them, to check that the LLM agrees with what you are saying. They'll even challenge people to prove some copilot output is incorrect. It seems to me that they consider LLMs more reliable than people.
Sounds like they would have been a fan of ungit [0], which I have seen used for that kind of flow overview, though it has looked more impressive than actually proved helpful in my experience.
you know, this seems very reasonable until that last sentence.
Which itself begs why they'd need this level of detail on the codebase.
i don't care how it's presented, via webpage or cli tool or whatever -- just saying that when you are working at larger scale, those are very reasonable things to want to see in one spot at a glance.
the need dissipates as you scale down.
I'm sure most monitoring systems have some way to loading that into their system.
If you really want to know this type of commit-level data you can get it from git pretty easily, even if you're not particularly good with git but can search half-decently. If you don't have the skills to use git, it's extremely unlikely that knowing what the current branch and commit status of the repository is will meaningfully help you do your job.
There's only one person you work with that's like that, right? Right?
They're definitely "leadership material"!
I tried to develop the skill of nodding and then doing it correctly later, but it would always hit because there was a ticket I had written getting assigned to another dev and I had to explain to them why it was the way I had specified, and then he would correct me, and I would say yes part of what you say is correct and needs to be considered (as I said, I think he was a good developer at one time) but not all of it and he would insist he was correct and I had to go talk it over later with the dev as to why it worked the way I specified.
But in the end, if you can't work with the manager, it's time to head to greener pastures.
Only their boss can reign them in, and so you have to use techniques to shine a light on them to their superiors.
Think "I want you to record your order" from HBO's Chernobyl, but more surreptitious.
Effectively a form of trust and reputation arbitrage, but it was effective for dealing with a particularly difficult manager who didn’t accept certain things about the design of an API, and yet when the other guy told him the same things, he just asked a few mild follow ups and accepted what I was telling him all along.
Dear lord, these tools have just come out, how can they already have invented a new type of asshole?
It would output completely nonsensical stuff like QR-Code formats that don't exist, or asking to connect to hallucinated APIs.
It was often caught by lead devs quite quickly: the documentation wasn't a link or a PDF but rather some block of text.
But in the cases it wasn't, it was super costly: some developer would spend hours trying to make the API work to no avail, or, in the case of the QR code, it would reach QA which would be puzzled about how to test it.
So yes there is a new type of asshole.
Not GP, but at my old job making games we had a truly horrendous asset import pipeline. A big chunk of it was a human going through a menial 30 minute process that was identical for each set of assets. I took on the task of automating it.
I made a CLI application that took a folder path, and then did the asset import for you. It was structured into "layers" basically, where the first layer would make sure all the files were present and correct any file names, the next layer would ensure that all of the textures were the right size, etc. etc.
This funneled the state of "any possible file tree" to "100% verified valid set of game assets", hence the funnel approach.
It didn't accept 100% of """valid""" inputs, but adding cases to handle ones I'd missed was pretty easy because the control flow was very straightforward. (lots of quotes on "valid" because what I thought should be acceptable v.s. what the people making the assets thought should be acceptable were very different)
Not the OP, but I worked on loans once. The application originally required tax returns to check income. Then we added the option to upload recent bank statements instead of tax returns. But there were a lot of places where the assumption of tax returns was hard-coded, so to save time the developers basically added an entirely separate code path for bank statements. Do this n times, and you have 2^n code paths.
When we eventually refactored and fixed it, we replaced the hardcoded references to tax returns with a concept of estimated income. Then we were able to reduce the branch to simply saying "if tax returns are present, estimated income = get_estimated_income_from_tax_returns(), otherwise estimated income = get_estimated_income_from_bank_statement()".
That's the core idea – collapse a branch as soon as possible.
Isn’t that the whole point of hiring experts?! So that you can ask them, instead of the computer for advice?
To give a contrived/trivial example, imagine a TLS handshake. Rather than building support to allow hosts to retry with a different cert, it's better to fail the connection and let the client start from scratch. Same principle can be applied to more complex process automation tasks in business. Imagine a leave tracking system. It might be better to not support changing dates of an existing leave application, and instead supporting cancel & re-apply. Best part is that the user facing part of both versions can be exactly the same.
With all due respect, this is preached by pretty much every book you read on cloud administration. I'd argue that if the process is decent enough, it'll work with major cloud providers, because their API's are rich enough to enable this already.
The thing with most automation tools though is, a) they're abysmal for most of the workflows preached (thinking of ansible and im shuddering) and b) to reach the degree of automation described in most literature, you need the API's of $MAJOR_CLOUD_PROVIDER.
I was an early eng and first VP of Product at Flexport. Global logistics is inherently complicated and involves coordinating many disparate parties. To complete any step in the workflow, you're generally taking in input data from a bunch of different companies, each of which have varying formats and quality of data. A very challenging context if your goal is process automation.
The only way to make progress was exactly the way you described. At each step of the workflow, you need to design at least 2 potential resolution pathways:
1. Automated
2. Manual
For the manual case, you have to actually build the interfaces for an operator to do the manual work and encode the results of their work as either:
1. Input into the automated step
2. Or, in the same format as the output of the automated case
In either case, this is precisely aligned with your "reuinifying divergent paths" framing.
In the automated case, you actually may wind up with N different automation pathways for each workflow step. For an example at Flexport: if we needed to ingest some information from an ocean carrier, we often had to build custom processors for each of the big carriers. And if the volume with a given trading partner didn't justify that investment, then it went to the manual case.
From the software engineering framing, it's not that different from building a micro-services architecture. You encapsulate complexity and expose standard inputs and outputs. This avoids creating an incomprehensible mess and also allows the work to be subdivided for individual teams to solve.
All that said – doing this in practice at a scaling organization is tough. The micro-services framing is hard to explain to people who haven't internalized the message.
But yeah, 100% automation is a wild-goose chase. Maybe you eventually get it, maybe not. But you have to start with the assumption that you won't or you never will.
For example, we integrated with a data source that used OCR to scan container numbers as they passed through various way points while they were on trains. The tech wasn't perfect. We frequently got reports from the rail data source that a train was, for example, passing through the middle of the country when we knew with 100% certainty that it was currently in the middle of the pacific ocean on a boat. That spurious data could be safely thrown out on logical grounds. Other cases were not as straightforward!
This is critical. So many people jump from "it works" to "let's automate it entirely" without understanding all the nuances, all the various conditions and intermediate states that exist in the real world. The result is a brittle system that is unreliable and requires constant manual intervention.
A better approach is to semi-automate things first. Write scripts with manual QA checkpoints and playbooks that get executed directly by devs. Do the thing yourself until it's so boring it hurts. Then automate it.
1. Only automate the steps you know how to execute manually very well. This is a prerequisite. This kind of goes in the direction of not automating everything but looking at the parts you can automate well first.
2. Enable human intervention in the automation. Enable humans to stop the line when something is wrong. Then you figure out what's wrong and improve your automation. This implies the interative approach.
[1] https://global.toyota/en/company/vision-and-philosophy/produ...
Only 100% trust the automation after every possible step, procedure, and word has been 100% verified. (Which is to say almost never…)
A whole family of problems across domains : supply chains, manufacturing lines, AI pipelines, resilient software, physical security, etc. come down to effective state limitation.
In my day job, the vast bulk of my arguments with PMs come down to a lack of planning allocation for post-launch code cleanup. I haven't been able to find a succinct articulation for the utility of 'code cleanup' to a wider audience. 'State-limitation' fits nicely. Exactly the concept I was looking for.
It draws from the better known 'less-is-more' adage, but adds a lot of implicit detail to the generic (almost cliche) adage.
So long as it's done silently, blended in with other things, and cloaked under clever wording (e.g. "this blocks that other thing you want" rather than "this will improve the codebase"), things will go quite well.
As soon as you speak to them as you would another engineer, you provide them material to use against you in prevention of you taking proper action.
If one person writes broken code in half the time, while you take twice as much cleaning the mess.....then you're going to be perceived as ineffective.
A fancy way of saying, “simplicity is the mark of truth.”
Or
“Less mechanical points of failure the better.”
I concur. Eliminate design risk.
My team and I once took on a very tricky automation project. At the time we had a complex software deployment done about once per month that involved a team of about a dozen people showing up at 4am to do it while traffic was low.
The deployment involved many manual steps and coordination of everybody involved. The person leading each deployment followed the documented list of steps and got each person to do their bit at the right time; people to run database migrations, people to install RPMs on particular servers, people to test and verify functionality. Mistakes and missed steps were not uncommon.
The very first thing we did was take the documentation and write a Jenkins job to post each step into a Slack channel specifically for coordinating the deployments. Someone clicked "go" and each step was posted as a message in that channel with a 'done' button to be clicked when that step was done. Clicking the button caused the next step to be posted.
The next release we did used that instead of one person reading the steps out of confluence. Everyone involved in the release could always see what step we were at, and when it was their turn to do their bit. This helped ensure no steps were ever missed too.
Over the following months we chipped away at that job a bit at a time. We'd pick a step in the process and automate just that step, starting with the low-hanging fruit first. The Slack message for that step went from "click to confirm you've done it" to "click to do it", with the result posted once it was done; followed by the next step to perform.
It was a long process, but it allowed the rest of the business (and us!) to gradually gain confidence in the automation, and lowered the risk of the project dramatically. Once several steps had been automated and battle-tested we removed the 'click to do' bits in between and the whole release became a couple of clicks followed by the odd bit of manual QA.
Was it hard to achieve buy in from all parties? I'd guess that would be the hardest part, to get everyone to join in on working on the automation.
> > https://blog.danslimmon.com/2019/07/15/do-nothing-scripting-...
What is the point of defining a Python class with a single `run` method, and then running with `Class.run()`, instead of just defining a `function` and running with `function()`?
Step 3: Go to this URL and do a thing, then click Y.
Do some stuff, do some more stuff
Step 8: Go to this URL and undo the thing you did in Step 3, then click Y.
An approach like this seems to give more weight to the fact that just figuring out a way to document exactly what needs to be done is often the hardest part and if you get that right before you start writing automation code, it might make make the automation much more efficient.
[1] https://blog.danslimmon.com/2019/07/15/do-nothing-scripting-...
Automating a process makes it more standardized and legible, but takes away a lot of the nuance and of the resilience that human institutions tend to bake in. Do it too little, you get chaos; do it too much, you're destroying the very thing you were trying to nurture.
It's certainly deformation professionnelle, but the parallel with software is eerily relevant.
It’s pretty tricky to get right and is very case by case in what it specifically means. But it’s been critical to handling edge cases.
For example, in 99% of cases, you can’t progress to stage X without doing Y. But in this case „I know what I’m doing”.
Though this puts a lot of pressure on downstream processes to handle missing data. Fortunately that hasn’t been an issue too much because validation grows and changes over time and must naturally deal with records that were created under prior validation when their state was originally valid.
One time I was involved in automating a business process. Nothing super complicated - scrapping data from a portal, filling forms, some data wrangling. It had to handle a few edge cases and involve a human in the loop. I tried to include as much data validation as possible, but this being a business in real world there were of course some exceptions. I implemented a Superuser Override which was basically a button that launched a popup with another form where the app users requested an approval from their manager to accept the unvalidated data. It was red and scary and it had Bootstrap .danger all over because it was a cool thing to do back then. Don't judge, ok?
Things worked as they should have. The automation saved tons of time, even if some fields were missing. Occasionally, there was a Superuser Override request which sent an email to the manager overseeing the process. Actually, things worked too well - the volume of processed data increased and so did the number of the Override requests. The manager - busy as they usually are - started approving all the requests. Manually at first and then they ran a macro that did it for them.
Eventually the app users got wind of it and they started using the Superuser Override liberally. Before they shown more restraint.
I wish there was some clever punch line to this story but I don't know what happened after.
If I had to do it again, I'd be more aggressive with validations, make the approval process more like a code review, involve the most experienced app users as the Superuser, and have audit logs. Oh and Tailwind instead of Bootstrap.
Audit logging is definitely another must for me. I literally won't make an app without it. The app I'm working on now had a case where a PM promised up and down that edit history wasn't needed for a particular section, and lo-and-behold 2 years later it turns out that actually we do need the history.
http://www.complexcognition.co.uk/2021/06/ironies-of-automat...
Being flexible is good, but it comes at a cost. When that cost is too high, don’t do it. Realize that the customer who wants that workflow probably isn’t going to be your make-or-break moment. And in fact, they might have flexibility of their own. For example, you don’t accept Amex but they have a backup credit card or cash. It might be annoying to them if they are fussy, but it’s normal enough that the consequences are minimal. And yes, you may occasionally get a customer who doesn’t have that flexibility, but you shouldn’t be pinning your business on rare events. Figure out what’s most common among your target customers and support a few simple workflows. Say no to everything else until you have the resources to do it properly.
Another thing you can do is have a hybrid low tech/high tech solution. Automate structured inputs with software. Write down the rest as notes in a logbook. Over time you will probably see patterns in the logbook that you can automate.
Lastly, remember what Morpheus says in The Matrix, “Some [rules] can be bent. Others can be broken.” For example, you could simply pay the bill on behalf of the customer who wants cash on delivery. Now you assume some personal risk but the computer system doesn’t have to support their workflow. Is it worth it? Maybe, maybe not, but it’s a choice you can make.
So much harder to say no that it sounds though - sales is saying yes to everything they can, board is pressuring for more sales, the users are taking every possible opportunity to shift blame onto the software, and engineering is buckling under the churn of changing business goals every quarter.
If you have a long term big customer who has always been able to do COD and you suddenly pull that option with the explanation "it's easier for us" that's not going to go over well. Now you might lose them and they will tell their network "Wow, Foo Corp used to be good now they are just being unreasonable."
Or just recognize that your software ideology is incorrect for the problem at hand. This guy wanted to write a single piece of software that did everything the business needed. That was clearly a mistake and no surprise that his software ended up very tightly coupled to the business itself.
This was an error in interface and component design. Most likely caused by starting to write software before fully understanding the job roles and people who would be using it.
the corollary of this, though, is that "there's nothing more permanent than a temporary fix". so, balancing these ideals is the work of engineering and management.
Are you selling to restaurants? To grocery stores in a large city's Chinatown? And so on.
How did you arrive at that conclusion? Most of those applications were written by the businesses themselves. IBM provided you a programming environment not a set of predefined business software. Which is why COBOL jobs still exist.
What changed with the mainframe was that instead of having a bunch of disparate and disconnected processes you now had a centralized set of dependent processes and a database that provided a single source of truth.
Businesses were going to want this capability regardless of how limited humans with green eyeshades were previously.
> Much of the early internet (and still most bank and insurance) look like HTML front ends to mainframe 3270 screens.
Well, precisely, those are custom applications. If they weren't we wouldn't have this issue. You talk about automation but you seem to have not noticed that mainframes have CICS, why it exists, and why businesses still use it.
The "old school" ways are actually still exceptionally powerful.
> As it turns out the entire flow at the post office (or DMV or tax office) is about exception handling. No amount of software is going to get you out of there because it is piecing together a bunch of inputs and outputs that are outside the bounds of a system.
Not sure how the author arrived at this conclusion, either. Both entities (USPS and DMV) have elaborate rules for how business gets done. The USPS in particular has rules that cover shipping everything imaginable, often via different methods. The outputs are quite simple: USPS ships your package; DMV provides your license or vehicle registration.
Some kinds of automation are absolutely essential. Aligning wafer stages during photolithography, shipping logistics, high frequency trading, etc. The business absolutely wouldn't work without it.
The other kinds of automation are more questionable. Developing E2E CI/CD automation for an internal tool that is redeployed once every quarter might be a more difficult sell to management. For these cases, even if the rate of manual process invocation is somewhat high (i.e. frustrating for some employees), the customer can't see it. They won't pay one additional cent for this.
There is also this entire meta time wasting game where the mere debate about "should we automate" takes up so much resources you could have manually invoked the thing more times than it would ever be feasibly invoked automatically during its projected lifetime. Alternatively, you could have just let your developers build the damn thing to get it out of their systems.
I've generally had the experience that it's the exact opposite. Best practices dictate that all processes should be automated to the limit.
Premature automation can create more complexity in the process as edge cases are usually added in, or needing routing.
If you're building process management and automation from scratch as a way to learn, it's also a red flag of technical debt waiting.
What happens if you just say no? I have a feeling a lot of complexity stems from the fact that these exceptions are easy to handle when you’re doing things manually, so it’s hard to say no. But if you think about the business case I would not be surprised if the correct answer is no more often than not.
Do you want to say no?
That's not an easy question to answer. You should think about it. Is it worth alienating some customers to handle them in a larger scale?
Naively, this looks like an automatic "no", but it's not.
Hardware automation has had obvious boons to places it works. And often we will force the places we want it to be to change to accommodate the automation. Agriculture is an easy example of what I mean here. Take a look at some fields that are optimized for giant machinery. Very impressive and can manage a lot. My favorite recently was seeing the "kickout" effort to prevent bad fruit from entering a hopper.
To that end, is the main issue with software automation that it thinks it can be a total solution and often tries to start by not imposing changes on what is being automated?
The part that is unique to software is that companies often expect people whose only expertise is in software to do both of these tasks when the second often requires deep domain knowledge. When one mechanises something in hardware it is generally taken for granted that domain experts will be central to the effort but when the result is principally software, domain experts are often left out of the process.
Public reaction to automation has been mixed for obvious reasons, but also because when software is applied to structure- and/or determinism-resistant systems it fails to capture some essential components that ends up degrading the services that system can provide.
It always was the biggest issue I have seen “sorry cannot help you with that because our system says so”, which should never be the case. There should always be way around the system. Then of course way around the system needs approval from some higher up anyways.
Another gem from the article that I wanted to surface. It's yet-another-take on the general sentiment here, but it's very succinct. Automation is good, but automation is a tool that serves people (and not in a Soylent Green kind of way).
I start making a tool for myself to track a tedious or error prone process. Even after you've gotten the wiki to have the correct steps in the correct order, you can still run into human error. I use it and work on it until I stop burning my fingers. Then I share it with a few people who complain about the same thing, and see how they burn their fingers.
Then I roll it out to a wider team, wait for more problems, then suggest it for general use. Usually by this point we are seeing fewer problems with the tools than with humans doing things manually, so we escalate to teasing people to use the tool so they don't end up with an Incident. Then it becomes mandatory and anyone who still refuses to use it is now risking PIP-level trouble if they don't join the party.
People who try to skip steps make not only a lot of problems for themselves but also for anyone else who suggests automation tools. Running toward the finish line makes everything worse.
Anyway, one other insight I would add is that the issues tend to come up at the interface between the systems. Your automation has to get some input and send some output and both those are pain points. That’s why we sometimes prefer imperfect monolith software.
Imagine test automation on physical devices (ex-founder of a mobile app testing SaaS company here) where system0level events that can't be programmatically detected or dismissed can delay, interrupt or alter the outcome of your tests.
Dealing with exceptions to make execution outcomes more stable seem like an obvious choice, but what if it's not feasible due to the complexity of the environment? We found that accepting variance and extrapolating test results from multiple executions in the same (or identical) environment was the only way forward.
Medical diagnosis is interpretation, you can't predict all possible states involved in the diagnosis and outcome, it is inherently unsuitable to be treated in such a way.
Stay Healthy!
For example, should an automation fail, will manual processes still work? Does execution of a manual correction, or repair, require the knowledge or skills of a particular person or persons that may not be available, etc, or can it be done by the normal persons that the automation is intended to serve/benefit?
In the Smart Home context, when the automation or other cleverness fails, will ____ still be operable normally by a regular person?