How I Automated My Job | Hacker News Reader

How I Automated My Job

medium.com

202 pointscodingcoop81 comments

Almost 10 years ago, when I just started to work, I had to do a repetitive task. Something in IIS about 30-50 subdomain bindings. I told to myself “I’m not going to do that manually, such a waste of time, let’s write a script!”. 5 hours later, I was still working on this script and I was stuck. I asked some help to a senior colleague and his reply was: “you know, it would have taken you only 10 min to do it manually”. :)

If it's a one time task then there is no need for a script, but if you have to do it again and again then it's very much worth it to write a script.

Scripts/automation can be worth it even for one-shot tasks for which time to create a script > time to do the task manually:

a) They help in applying forethought to a process. Maybe you can even dry-run the thing.

b) They help documenting what has been done, and how, which can be invaluable in a team setting.

c) They train you into getting incrementally better at automating, and at sensing when to automate and when not to†. If it's painful, you're not doing enough of it.

† I find that for the ones that would take ages to properly script, more often than not a combination of manual (like preparing a normalised input for the script using vim) and automated subtasks can do wonders, leveraging the best of wetware and software worlds.

The only thing that I will do manually in any environment is putting sensitive data into configuration. With IAM roles, I can usually avoid that. I will create the roles manually.

If you go by the old mantra of treating servers like cattle and not pets, you have to automate everything. My VMs (EC2 instances) can be destroyed, replicated, auto scaled, etc. at any time. I have to automate no matter how long it takes to figure out.

You are so right from a business standpoint. But what about personal development? This is one of the ways developers are born. Sure the first one or ten attempts to automate something might be a net loss to the business, but eventually that junior employee has leveled up their programming skills and developed an estimating sense of when to automate and when to suck it up. It's easy to sit at our place of knowledge and tut at the beginers but we were all beginers once.

I'm an "early" programmer and still poking my way around various toy side projects to learn concepts and such.

Ubuntu and Bash had been somewhat of a mystery to me previously, but I eventually got a virtual dev environment setup and was comfortable getting in with navigating to my Vagrant folder, then executing 'vagrant up', 'vagrant ssh', etc. and then navigating to my project.

That comfort turned to annoyance as I realized how unnecessary those few keystrokes were. And despite how little time they took, I found myself getting angry at the repetition of having to enter them.

So I decided to conquer my Bash scripting fears, and wrote what many would consider to be a trivial script to navigate to the Vagrant folder on my host machine, boot up the VM, and then navigate to the appropriate project folder in the VM. Along the way I learned more about aliases, and also how to pass commands from a host to a guest machine which was one of the roadblocks I encountered.

I now can boot up and get into my project folder with a simple three character alias from within any directory on my host machine.

Again--any experienced developer would be able to script this in minutes most likely. Back then it took hours. And now those are the most satisfying three characters I type, as every time I enter them, I get reminded of the progress I'm making and that even trivial things may be candidates for automation.

You could still save yourself the misery of typing one or two characters if you fix that alias.

you just blew my mind. im not even kidding.

Doesn't every developer eventually alias building to 'b' or maybe 'm'? It's like a law of nature. Or you can just rebuild automatically every time you save a file, like js people do.

I know, right? I'm actually leaning in the direction of making it a little less spartan by having it either accept a parameter for a project name to go directly to that folder, or prompt me to enter a number to select from a list of projects gathered from the folder names in my parent project directory.

Just in most cases its not necessarily better to have written the script. You could learn the script in a situation in which the time payoff isnt so terrible.

How can you possibly estimate the time payoff if you've never done it before?

In any case, I think it's more than just the time payoff. Getting repetitive tasks automated allows you to reallocate that cognitive overhead to other details, and that's a payoff that's hard to quantify, but is critical for leaving your mind free to focus on more important complexity.

Quite often you have to manually do the task at least once to determine how to automate it. That tells you your upper bound on how much time can be saved.

If for example it only took 15 minutes of manual work it's highly unlikely that the payback of automation would occur any time soon. That's especially true because the 2nd or 3rd time there may be different special cases and you'd have to go back and change your automation.

Automation's payoffs aren't necessarily in time saved. Quality can be a more important consideration. "Did I forget a step?" "Did I forget to verify this aspect of the configuration?" "Oh ... am I sure I typed in the new domain name correctly each of the eight times I needed to type it in in the various contexts?"

Automating a task usually means each important parameter is entered only once, that all steps are completed, and that the results are checked.

Of course if we agree programming is not so easy, verifying that the script actually works as intended will take roughly the same time. Not to mention thinking about what it should do when faced with an unusual situation.

Automation can still be a benefit even if it initially takes more time. People doing repetitive tasks make mistakes.

Spend 10 hours writing a script to automate 50x10 minute jobs. You spend an extra hour, but you end up with 50 consistent results, rather than 40 successful jobs and 10 with minor mistakes.

From a short time perspective you wasted your time. But in the long run you learned valuable skills that later may have enabled you to automate other more important things.

From a pure time savings perspective, this makes sense - naturally, links to XKCD are already popping up.

From a maintenance perspective, a well-written script can serve as a reference after the original task was long forgotten. A poor substitute for documentation, but when performing software archaeology on legacy systems you really need everything you can get.

Often the time invested in writing an automation script far outweighs the time spent doing it manually, even when you account for every time the script will ever be run. And it can still be worth it if the script never makes a mistake that human might.

even when it doesn't. I find that if you are careful about adhering to the unix philosophy of tiny scripts that do one thing well then you end up with lots of tools to solve problems specific to your problem domain that get reused in future tools.

This xkcd is really relevant, being a chart of the time invested vs time saved, not even including the accuracy and learning value: https://xkcd.com/1205/

I always look at it as decreasing a dependency on me personally. If I'm sick, the task can't be done at all, for example. So even if you spend an hour automating something that can be done manually in 10 minutes, it's a good use of time because the script doesn't get sick, doesn't go on vacation, doesn't leave for another company, etc.

The flip side of this is your company gradually becoming dependent on a sprawling and ever-growing tangle of likely undocumented scripts. This can be a time bomb (no matter how good your intentions) waiting to explode if you ever leave.

Not necessarily. If the script is documented and written cleanly (e.g Fabric3 which has very little magic, and reads almost like Bash), then in can server as "runnable documentation". In this case automation could be really good for continuity and busfactor.

But yeah... got to put some effort into it. Like all docs, script-docs worn't write themselves.

Agree that it's not unavoidable, but the problem is deeper than just the scripts themselves being well documented and straightforward. The main risk (IME, anyway) is awareness / discovery of the scripts in a timely manner - the fact that everything 'just works' means that it can be months or years before something breaks, at which point it may be too late to ask the person who set it up. Hopefully there's good handover from the outgoing staff member to a technically proficient colleague. If so then everything might be OK but if not then a catastrophic failure may well be the first sign that something is wrong.

(A recent example of much simpler automation going horribly wrong was a company losing their top-level domain name because it was set to auto-renew using an ex-employee's personal credit card, and their email address as contact. This worked for years until that card was cancelled and the warning emails bounced off the closed account.)

I disagree. If you make a mess, you have a mess... but at least your intentions were at one point clear enough to encode as a computer program. While painful to pick apart during an emergency, it's easier than finding the one person in the world with that knowledge and getting them to explain it to you. Having your business depend on a messy pile of scripts is better than having your business depend on someone who died a few months ago. Having your business depend on clean well-maintained code is even better... but great is the enemy of good.

I can think of a few benefits, even if it takes longer: you may need to do something similar again and now you have the script, you've gained experience and you might right the next script faster, correctness.

I think correctness is the most important one. Even if a manual task is quick, ensuring that you do it correctly and exactly the same way for all 30-50 things is extremely tedious. If one out of those 50 subdomains was behaving differently and you needed to investigate exactly why, that could quickly erode any time savings.

Correctness is huge. The other day a colleague and I were figuring out some things about scheduled work days over the next few years. After I had futzed about in a spreadsheet for five minutes, he said "this is ridiculous!" and started furiously paging through a paper calendar. We had our results nearly simultaneously, but he was off by over a month. After that we just used the spreadsheet.

Correctness also works the other way around. We needed to link to databases with non-matching customer IDs, the analyst on the team spent several hours automating fuzzy matching of customer names in R before we concluded it was easier and more correct to build a mapping table by hand for the few 100 that didn’t match.

My rationale is that every time I automate a task, I get better at automating tasks. Losing a battle here and there helps me win more battles later on, and by the end of my life I will have netted more time saved than lost.

Also, automation work is much more interesting than doing stupid, repetitive tasks.

I am reminded of this Brian Marick classic:

http://www.exampler.com/testing-com/writings/automate.pdf

Relevant : https://xkcd.com/1319/ and : https://xkcd.com/1205/

sure, but mess it up once and you spent time to figure the error and fix it.

or go on holidays and your colleagues just gotta run the script.

the big benefits in scripts that is often overlooked: Given the same conditions and input, the output will always meet your expectations. There's no chance for human error.

A classic. Someone else might have said: next script will only take 15min.

Maybe partial automation would have been a good roi.

Or maybe being deep into manual is also neat for your brain.

For data analysis, there’s a lot to be said for examining the data instead of blindly piping your data through something.

I make these tradeoffs a lot. Automating is fun but time consuming. Doing things manually is tedious. However, automating has one benefit: you make things easier for the next time you do something similar because you learn something. Usually requests for tedious things are not isolated incidents. So I have a bunch of commandline stuff that I know how to glue together like csvkit, jq, xargs, and a simple bash script that adds functions to do rest against our api server. Using that I can do easy bulk operations against our db without investing in product features, ui, etc.

Upfront time investment to build automation script sometimes more than manual. But most of the time it's a one time effort to save more of your future time.

Terrance Parr's motto (the creator of ANTLR) is “Why program by hand in five days what you can spend five years of your life automating?”

XKCD always delivers: https://xkcd.com/1205/

I am a script junkie. I have to, just have to script every task out. Here's what I've realized.

First do it manually, then automate it.

Doing manual tasks is frustrating enough to motivate me to automate it. Plus I have a much better understanding of the inputs, the required outputs and constraints for the script to be more useful.

Let your scripts evolve.

Don't force everything in the first go. Simplicity is the key. Let them be rigid, readable and in a single file at first. Then, introduce flexibility as required. Abstract out core functions into individual scripts/functions/modules. This builds a certain sense of familiarity with the code that is just too endearing and eventually very productive.

Reuse your mini-scripts.

Each of my scripts does one thing. There are super-scripts that combine many of those for a task that I am performing at the moment. DRY principle. The more simple, atomic and mutually agreeable your scripts are the more powerful they get. This is just the unix way of doing things. Nothing new.

Write consistent code comments

You will write. You will forget. Make the code comments personal and consistent enough for you to immediately recognize them.

Use version control

Absolutely use it. There will be many versions of your core scripts/functions that dont agree with each other. My personal preference is a version number (semver) suffix on each file to identify which core script is used by which task script.

Reap the rewards

The rewards of putting in the effort to me have been akin to compound interest on an early investment. I have reused core scripts so heavily that prototyping any new idea is pretty easy. If there is something that a script does not do, I know what I need to read over the next weekend :)

I like what the guy did, but am I the only one who feels most of these "I did x with y" articles always end up trying to sell you something? What happened to just sharing our knowledge with each other?

Most content on the internet is created for marketing purposes, unfortunately. I don't have much room to talk, as I do it myself. That doesn't mean that we don't enjoy creating the content, or sharing our knowledge with others, but it can definitely get old being the consumer of these pieces.

I’m not sure about that. How do you measure? I would expect that the vast quantity of content created is not to sell, its by individual creators writing these comments, tweeting, blogging, etc.

Perhaps the majority of content read, but that’s because it is marketed.

I’m not sure how you would measure. Count all the bytes on .com vs .org?

> Most content on the internet is created for marketing purposes, unfortunately

Why is this unfortunate?

Because the marketing is incredibly low quality. It's equivalent of most mail I get. Completely ignored.

Well he doesn't have a job now, does he? ;)

This article goes through discussing using tjs commander library for building node cli’s. I’m not a big fan of the library.

A recent approach I’ve been taking is just parsing args with something like minimist or zeit/arg and then handling the commands passed in.

It leads to a lot simpler code especially for small tasks like what is described in the parent article.

Another tip, you could also use pkg to create binaries for your new cli application so people don’t need node installed on their machine to run them.

I'm curious as to what you mean by simpler code?

Commander leads to incredibly simple code (as it just sets a list of options) combined with helpful CLI help output.

Some of the meaning of the code is embedded in how commander is set up, which if one is unfamiliar with it means the code is less explicit. Explicit is good, and there is a sense in which explicitness leads to simplicity.

Exactly, it’s a lot clearer to your future self, and any other maintainers how the application works when you aren’t adhering to a “frameworks” way of doing something. Especially for a simple cli.

Abstractions can be good, but I feel as though something like commander is really just an unneccessary abstraction.

I’ll take simplicity first anyday.

It’s extren explicit...? Each option is laid out clearly. Each option Maps to a key on the object that is passed to the function called at the end of a commander chain.

It would be hard to be more explicit

Link to a post on ACM Queue about the benefits and ramifications of automating as a habit: https://queue.acm.org/detail.cfm?id=3197520

Tom L is a good writer - I have gotten a lot out of his books and articles over the years - excellent point here:

"Since February 2015, the SRE (site reliability engineering) team at Stack Overflow has switched from a mixture of Python and Bash to Go. Even though Go isn't a scripting language, for small programs it compiles and runs nearly as fast as Python takes to start. At Stack Overflow we tend to prefer compiled, type-checked languages for large programs, especially when multiple people are collaborating, and, therefore, no one person is familiar with every line of code. Our policy was that Bash scripts couldn't be larger than 100 lines and Python programs couldn't be larger than 1,000 lines. Those seemed like reasonable limits. Rewriting scripts when they grew beyond the limit, however, was a lot of work. It was better to start in Go and avoid the conversion."

Not Golang, per se, but over time you are limited by your tools. Big bash scripts get unwieldy. Nodejs changes underlying dependencies (coincidentally wrestling with 4 yr old nodejs this week).

Reminds me of this:

https://www.techworm.net/2016/06/programmer-automates-job-si...

That's an article summarizing a reddit comment but splits it up.

This article is slightly better as it links to the original advice thread:

https://boingboing.net/2016/06/08/coder-fired-after-6-years-...

That link eventually led me to this, which is one of the most enjoyable things I have read recently:

American Dream: https://github.com/bibanon/bibanon/blob/master/Stories/Ameri...

That's brilliant.

Wow, I had read a first generation of that piece a long while back. It ended right after the call to the guy with the same job code. It has grown!

What I noticed about this - I'm not thrilled right now about using node to write console helper scripts, but the CLI lib they're using here does have a pretty nice interface. I do like Python generally for doing scripting type things, but there doesn't seem to be any mindshare for any cli helper libs for it other than click, which IMO has a really clumsy, awkward interface. Ruby is generally my favorite go-to now, and it has a bunch of cli helper libs with a variety of types of interfaces, at least some of which have what I consider to be a really slick interface.

If I was better at Python, I might try my hand at writing a better Python cli lib myself. I'm surprised nobody else has yet.

Heroku has released their cli framework recently. It’s really good.

I have had some nice results with swift. It’s still predominately Apple tech (macOS,iOS) but the packages manager is nice and it’s has nice “script” feel for a compiled language.

Not sure what you are referring to, Py CLI libs are a dime a dozen? Not to mention argparse has been getting better.

https://opensource.com/article/17/5/4-practical-python-libra...

I'm interested in finding a Python CLI lib for building command suite apps. I've written a few with Click, and its architecture seems weird and clumsy to me. I looked at that website, and it doesn't list any command suite libs other than Click. There's a few other CLI-related libs, but for doing stuff like making prompts work nicely.

My usual go-to for that kind of stuff is Escort for Ruby. Everything is defined at a place and level that makes sense IMO.

Searched for "command suite" got nothing. Searched for "ruby escort," found services in las vegas. ;)

If you mean sub-commands like version control software has, argparse can do it. A bit clunky, but doable.

In respect of some of the comments elsewhere saying that the person would probably have been better off just doing the damn job, XKCD has, of course, got there first.

https://xkcd.com/1205/

https://xkcd.com/1319/

I reference these regularly to keep me on track doing things that actually matter, rather than tinkering endlessly to save me having to do things that are less interesting, but would take less time.

Funny, I have those exact two comics sitting right outside my cubicle. #1319 was put there because my team lead thought I wrote too many scripts to automate silly stuff. I put #1205 there beside it so that I could rationalize all the automation I do.

Wow... these two comics should really come on the same page. Great job pairing them!

"automate" comes from "self" and "mating".

Automating comes from the roots "auto-" meaning "self-", and "mating", meaning "screwing"

FTFY

On the subject of automation, has anyone been able to do or is there even any scope for script level automation for video post-production? I'm talking specifically Adobe Premiere. I have to constantly do a series of the same actions on different content assets much like re-skinning. If anyone has had any success - I'd be interested to learn from you.

I have been looking into that of late and have found that the what little scripting options Adobe provides are woefully inadequate for anything more than basic macros.

I have been looking at FFMPEG for automating video post-production, and recommend you do the same.

What I want to know is - how can I automate the Google Admin interface. Because holy crap, is it slow.

Yeah, I'm sure there is an API :-) I just haven't gotten around to looking into it yet, it's one of those annoying things that never gets to the point of seriousness, but gets a few choice words from me when I have to use Google Admin.

Fortunately, much of GSuite admin api functionality is nicely wrapped up in the free / open / well maintained GAM cli - https://github.com/jay0lee/GAM/wiki

I have a simple rule. If I find my self doing something for the third time, automate it.

“Never send a Human to do a Machine’s job” - Agent Smith, The Matrix.

> The Bash scripts were good, but if someone was working on a Windows machine, they couldn’t be run

Git Bash. The only requirement is installing Git For Windows, which is an easy install.

Better yet, Windows Subsystem for Linux: https://docs.microsoft.com/en-us/windows/wsl/install-win10

oh look, an add pushing a book thinly veiled as an informative blog post.

how original. Saw this on reddit earlier this week, and didn't read it then, either.

I honestly refer to this whenever I want to automate a task:

https://xkcd.com/1205/

And also this for the lulz

https://xkcd.com/1319/

Perl 42

Reply on news.ycombinator.com