Manual Work is a Bug – Always be automating (2018)
queue.acm.org
queue.acm.org
If you automate anything you need robust error reporting (which is not an email someone will ignore).
Every developer should take this advice to heart. If a customer asks for 'a daily email report', you should strongly advise against it.
- Most reports the customer asks for will be meaningless vanity metrics anyway
- If you only do error reporting (not sending an email if everything goes smoothly), you won't notice it if email is not being delivered.
- If you report everything (not just the errors), people will stop reading them
- A mailbox history is not a log
- you'll get a request just about every week to change the recipient list of the report.
- Who reads the reports on weekends?
I could go on with arguments for a while, but the short story is this: email is not suitable for logging and reporting. In fact: email is not suitable for a lot of things. But customers will ask you anyway, because that's the tool they know (if you have a hammer, all problems will look like nails).
My advice:
- Log everything to disk when possible (syslog, pipe)
- use a logging aggregator to filter and archive (fluentd, ELK, graylog)
- use an exception tracker for exceptions (sentry, raygun)
- make incidents actionable by creating tickets automatically (jira, zendesk, slack)
- if needed, use an incident response service (pagerduty, opsgenie)
not really a serious concern, but what happens when ticket automation goes down? If all the problems are auto-reported, in a form of tickets or tasks, then if that somehow breaks down it might take a moment before anyone notices, especially if reported exceptions are rather rare. If it is normal to have tens of tickets every day, then problems with ticket creation will be noticeable almost instantly, if they are very rare - then not so much
The second issue with automated exception tracking is that you loose the "huh, this is weird" mechanism that works when actual humans go through logs or reports. While any tool will of course be orders of magnitude faster and also probably more accurate, by relying solely on such automation an opportunity to notice some "weird"/not-typical entries or rare/unexpected sequences of those might be missed. Then again in most cases - I guess - simple statistical analysis might be a good substitute. And that can be automated. (edit: formatting)
Who’ll watch the watchmen?
Also, the cost savings by automating the task should outweigh the monthly cost of the tools that are required to run it. If not, the task is probably not worth automating.
In political/economic debates, people like to say: "if you tax something, you get less of it." And in turn "if you subsidize something, you get more of it."
You could think of automation as a labor subsidy. Or, on a rough level, as a tax on attention. That's not strictly correct, but it gets you in the right frame of mind: automation is, among other things, designed the achieve labor with diminished (or even absent) attention.
The problem is that diminished attention can have overlooked tradeoffs. If you know those tradeoffs, you may be able to automate around them. If you're not paying attention, you probably can't. If you've automated away attention, you might not notice until you've hit a problem at the scale of your automation.
Automation is valuable, absolutely. You probably want more of it. I do. And yet...
The second time I do something o make a checklist, at a certain point I decide I need to automate it and I have all the steps mapped out to start from.
It works well on practice because a checklist is quite mutable, if something goes awry it gets added as a step the next time until I've covered the edge cases.
To make them I use vscode, markdown/journal plugin and mdpdf to print them to pdf, so they can be source controlled easily as well.
But then the IT department comes along and hates it; they make a wiki page for each server with a list of instructions for what was done. Their approach is more flexible, as not every tool/step can be easily or robustly automated, and if you can just screenshot the config webpage and paste it into your wiki, that might be just as good as (and a lot faster than) spending an hour trying to bypass that step.
Plus, their approach assumes aggressive use of VM-level tools. Where I'm like "oh, I need N build slaves? Let me add them to my inventory file and re-run", whereas they're like "N build slaves? Sure, I'll set up the first one manually and the clone the machine for the others."
I know there are real actual advantages to version controlled configuration, but there is something pragmatic about just doing the thing and keeping human-readable notes as you do it.
I learned a lot from my coworkers that I couldn't have learned at school or internet. The author of the comment looked like she is passionate for programming, so I think she could be much happier with coworkers who she can look up to and learn from.
Too much vs. not enough are just different sets of problems.
Too much = you risk no one on the team being able to troubleshoot when things go wrong because no one has ever had to run through the process themselves before so no one understands it.
Too little = a non-trivial percentage of your bandwidth is eaten up by trivial things you keep doing over and over again that are in the vain of servicing requests rather than solving problems.
In practice I find I'd rather have the first problem and I can think of solutions to it's problems that I'm happy enough with.
Sam Altman covers the same sentiment in http://blog.samaltman.com/how-to-be-successful .
[1]:https://www.xkcd.com/1319/
(I may have been doing a lot of automation recently because, I think, bosses have paid a lot of money for an automation tool and feel the need to use it, and therefore feeling a little cynical about the whole thing)
Some problems I struggle with:
1. How do I automate in an environment of so many disparate systems. Some things are done through web interfaces, some RDP'ing to an AD server, email, spreadsheets sitting on shared drives, etc.
2. How do I handle credentials. Central IT won't create generic accounts for us, even for testing, so all automation would have to be done through my credentials. How do I set up an automated system that uses my credentials safely?
3. Where do I run this system from? From my own laptop, a box under my desk, somewhere in the cloud where I'd have to beg security to allow it to do anything non-trivial? And how do I pass this on to the next person?
I've looked in to RPA type solutions (expensive), headless browsers (so unreliable), Microsoft Flow (limited and confusing).
I'm at a loss but I did recently find TagUI which looks very interesting https://github.com/kelaberetiv/TagUI
1. Using disparate tools if need be. Fortunately we have PowerShell these days which covers a whole lot of stuff including being ok at various web-based interactions. Anymore, if it can't be done with PowerShell (in a Windows environment that is) then I think it's probably too fragile to leave to automation anyway... ok, that's not entirely true. There is at least one important production system in our environment I have to automate using AutoHotKey and that works pretty well.
2. I have no idea why anyone would think not allowing generic accounts for this sort of thing would be a good idea, but under your own account shouldn't be a big deal as long as you leave appropriate documentation about what needs to happen in case you're hit by a bus. Many sins are forgivable if there is adequate documentation.
3. I usually just run automation scripts on the system I'm scripting. If I'm dropped into an environment and asked to figure out how some system is automated, where's the first place I'm going to look?
To help make the case to change the system, though, you could build a proof-of-concept that uses your credentials and present that - and the ACM Queue article! - to the folks you need to convince. Try showing it to someone who pays the bills too - they might be a useful ally to help change how things work if they recognise the productivity gains.
It still is a win because you’ll have a better chance on diagnosing and improving something that fails the same way as opposed to some manual process that has been done in a dozen different incorrect ways and you learn nothing from mistakes because there are no automated controls to feed back into.
Most people and organizations have a strong negative gut reaction when automation does something wrong but are okay when a manual process yields mistakes because we largely accept humans as fallible (at most, someone gets fired, and more bureaucracy is introduced), but the long-term economics of automation look ultimately better.
I’ve seen time and time again apparently innocent manual processes that, when not recognized as a liability, rapidly turn into entire departments (and entire management hierarchies), at which point it’s impossible to improve, because changing culture is harder than changing code. Having enough instances of that is what creates corporate whales without competitiveness.
What we have found helped us quite a bit is, after writing the initial runbook for a task, to write a second script (usually pretty short, maybe two dozen lines) that asks you questions and then generates customised steps. We then later use these functions in the actual automation.
> Do-nothing scripting: the key to gradual automation
The reason is simple: I manage systems not installed by me, where problems are often a one-off issue and more important, every system is installed a bit differently (differently enough that there is no portability for automation between systems).
Automation is awesome. But sometimes it just cannot be done, because of constraints put in place by others.
"'Automating' comes from the roots 'auto-' meaning 'self-', and 'mating', meaning 'screwing'."