Don't spend 6mins doing it by hand when you can spend 6h failing to automate it | Hacker News Reader

Don't spend 6mins doing it by hand when you can spend 6h failing to automate it

twitter.com

In a daily grind of frameworks, patterns, boilerplate and early optimization, I find automation work a lot more like the hacking I started with. It's a more rewarding and... joyful kind of programming.

Maybe it saves me time, and maybe it doesn't. Perhaps it makes tasks more repeatable. Mostly it makes me less likely to drop out and switch to a career in beach combing.

A few year back, an intern on my team was tasked with updating all the instances where we used a certain type of builder function with a replacement that we had written and wanted to encourage our users to use by example. My boss highly encouraged him to write a script to do that task rather than doing it manually, with the argument that even if wasn't any faster overall, it would make the task much less boring that it otherwise would be. That moment stuck with me a lot; a lot of times, we're highly encouraged to be as efficient as possible rather than doing things a way that might be a bit faster but is much more frustrating or boring, and yet we also have a systemic problem of burnout in our industry. Personally, as long as the end result will be the same, I find doing things in a way that will keep me engaged and energetic increases my productivity in the long run, as making the mundane tasks a bit more fun even at the cost of making them take a bit longer will end up making me much more ready to deal with the harder tasks with a mindset that will end up winning back the time lost and then some. (Note that I'm _not_ necessarily talking about something like using a library or framework that could bloat or slow down the final product, since the "end result" is not the same in that case, in which case the difference in the end result needs to be weighed against the potential benefits)

The other reason for automating a repetitive task is to reduce errors. When you do the same think 100 times you're bound to miss an instance, or paste the text in in the wrong place, or whatnot.

But rewriting code isn't always a repetitive task. If you're trying it with anything other than a full parser and refactoring toolkit, you risk breaking the code in ways that you don't realize because you're not actually looking.

If you really have written the same code many times, you might be able to get away with a search-and-replace approach, and with a little luck everything you break will get caught by a compiler. But the more different ways you wrote it, the more likely you are to make things worse.

Learning a refactoring toolkit might be a fun project, but it may well take more than just a few hours. Code-that-writes-code is pretty tricky.

As you say, the alternative is also risky. So getting it right may be worth the effort. But in a lot of cases, it will be quite a bit of effort.

It need not be code; it could be deleting the phone numbers from a plaintext database. Or any number of manual one-time tasks on data.

Yeah, I think you're right, keeping people motivated vs maximising productivity is key to get right.

There's also the angle of keeping your eye in with other languages, e.g learning or keeping your python skills up to date if you need to do a quick bit of scripting to automate a task (and you don't usually use python).

Now come clean and tell us how far down the path of becoming a beach comber you've considered

I live in the mountains and watch sailing videos every day.

Huh. So do I. Now I'm doing something about it. Heading down to Guaymas to take two ASA courses next week.

I posit there is a hackerish view of sailing, especially bigger boats. Complex systems with a lot of dependencies, and eg the weather gods to contend with.

Sailing looks posh from a distance, but it's a HARD problem, and if you don't succeed you could die. There are physical and mental struggles, endurance, knowledge, judgement. But the challenges are discrete, and all contained within a smallish box. And you can upgrade your box like an arcade racer. There's jargon! So much jargon! And there are sunsets and beaches in the bikini latitudes. And there's travel to new places, new foods and new people to meet. It looks like an amazing balance of challenge-and-reward, if you can afford it.

Enjoy your courses. I'm envious.

Beach combing is a career? Hmmm, very tempting.

There's also the matter of eliminating errors.

There's one script I wrote to partially automate posting podcasts. (I really should add a couple of additional steps.) It's almost certainly saved me time over time. But it also automates a fiddly process that's very easy to get something wrong if I do it by hand.

Do-nothing scripts are great for that too, I think this was shared here a while back

https://blog.danslimmon.com/2019/07/15/do-nothing-scripting-...

Nice. I like that idea.

Yeah that's the thing - about half the time I set out to automate something, I wind up in that

"Oh shit, this is taking longer than I thought -- so I'm just going to have to park it in a semi-finished state, and get back to it later. Grand, now I have two problems."

place.

>There's also the matter of eliminating errors.

Alternatively, you could replicate those errors at scale....

Automation becomes a type of documentation for certain tasks. Sure, a real, human-intended document might be better, but this script contains even more details about the process and might even be able to do some of the steps for you!

This is always the big thing to me - the crisp repeatability of an automated process often functions as it's own kind of executable documentation.

Opposite perhaps clarifying advice:

Do it by hand at least once or twice before trying to automate it.

Doing it by hand often teaches unexpected lessons and can shorten the automation effort.

I have seen far too often the jump from never done a thing to preparing to do a thing a million times a day.

I am not sure this is opposite advice, really. I make myself do an ops thing by hand, for a period of time. Eventually the repetitive parts begin to really annoy me. Then I automate the repetitive parts.

But I am deeply lazy. My father made me count fish hooks by hand from bulk into those dozen boxes for his fishing tackle wholesale business. I was, maybe 8? Couple of cents per box. Couldn't stand it. So I eyeballed batch sizes, and made mistakes. Should've weighed them. Boy did he get mad. But nope, no way I was going to count every one of those little hooks.

That's the way I feel about the repetitive parts of ops. It takes a while to figure out the repetitive bits though, and abstract those correctly.

I just spent 2 full days automating a 15 minute GIS task. I don't regret it, but 2 full days of cost was rough. I was stubborn and determined to do it. It could easily have ended in failure after 10 hours. The hardest part is figuring out what tools/libraries to use to do the automating. The actual coding is easy as hell.

The end result though, is beautiful. It's cleaner, more precise, more technically correct, and more consistent than what I was producing before. It makes what was already one of the most profitable parts of my business even more profitable. One of the big side benefits is I gained a lot more knowledge in the process that I've already been able to apply in other contexts.

So... it can be really painful and time-consuming to automate, but there are more benefits than just the future time savings.

A company I worked at used a home-rolled task tracker. It had data modeling for a Job to be done, and within it a series of Steps that had to be done in order. Users could select the steps they wished to view from a big list, and be presented with the work items in that queue sorted by date. That was it.

This task tracker worked fine for a period of time, until the list of Jobs exploded. The company added dozens of different types of tasks people could do, correlating to different products that had different delivery dates. Every person in the department had to be aware of every kind of sub-task they would be responsible for, and they had to manually subscribe to that sub-task or work would get lost. People spent a large amount of time every day selecting and un-selecting tasks just so they could view a focused list of todos, especially anyone in a shared role like Admins, or Lead Designer (who would also complete Design QA steps for other designers).

Having to do this multiple times a day really irritated me. I didn't have access to the codebase and management wasn't whitelisting any extra development, so I hacked around the problem. I discovered that sub-task subscriptions were stored in a cookie on the users' machine, so I built a JS Bookmarklet that would read your cookie and help you create a new bookmarklet you could save to your bookmarks bar. This new bookmarklet would automatically navigate you to the task tracker, switch out your list of subscribed sub-tasks, and reload the page. I made a simple HTML help page instructing users how to use the tool, and I provided a basic set of pre-baked bookmarklets to get people started. Simple stuff like, "Website Development Approvals" or "Templates Ready for Build" or "Work Order ready for Design".

This automation _exploded_. I instantly saved 20 minutes a day across 20+ people. But more importantly, this simple bit of code introduced and formalized the idea of a "Role Queue" at the company. Business Admins had confidence they weren't missing work items, because they all collaborated on maintaining an authoritative set of bookmarklets for their work. Same for Designers, QA, Developers, etc. This in turn changed how the departments were run. People could work off the "same" queues, tag team work, and make group decisions on priority.

Where before everyone saw a slightly different lists of work and felt different levels of stress, now there was consensus between coworkers. I saved thousands of man-hours, and really smoothed out the path to production for a new product, all because at its core, I resented having to do such a stupid task like "just click these boxes and uncheck these and save" just to go from viewing "Ad buildout" to "Website buildout"

Funny thing, the company is still using my bookmarklet code. Every now and then I get a text from an old friend or two working there laughing about the code comments I put into my core bookmarklet builder.

    /**
     * BetterQueues by Titus Johnson
     * This is hack to make it easier to view work queues
     * If you're reading this and this code is still in use, you need to get a better job
     */

You’re quite a writer, and the comment at the end made me spit out my coffee laughing.

You’ve made my day.

Don't forget the investment in learning. Maybe next time a similar task will take much less time.

The first time I used awk I struggled and probably took way longer than necessary, but the second time a similar task (process a bunch of columnar data and extract certain values) took no time.

Worth it.

That's what I always tell myself, except by the time I need to redo a similar task, months have passed and I have to re-read the documentation again.

Yeah, I have/had the same problem. I've started using spaced repetition to retain what I've done.

Check this out, spaced repetition for learning how to code: https://www.youtube.com/watch?v=kshXDo8psj8

Essentially, every time you encounter friction with something you write it down in Anki.

Anki then takes care of presenting the information to you based on how well you're able to recall.

After confusing Array.prototype.slice() and Array.prototype.splice() one too many times, I finally added it to Anki. I don't confuse them anymore.

Thanks for the recommendation, this is exactly the type of things I need. It suspect it will even be useful for my music theory learning!

Basically how I got my career underway. Here take over this process to manually crunch these numbers for 7 hours a day.

Couple weeks later after learning python, okay those take 5 minutes of human time to run. I made it a point to always automate regardless of time saved/wasted.

Paid off big for my career.

I learned this after I purchased a CNC for woodworking. A CNC is an amazing tool for so many jobs - and a horrible one for others. For some tasks, despite seeming repetitive and CNCable, manual woodworking tools speed up the process and the quality of the job.

An example: Adding chamfers to cutting boards. Sure, this could be done with a 45 degree bit in the CNC, but any deviation in board width/height/thickness means reprogramming the machine, and you can chamfer with a palm router in the time it takes just to affix the board to the CNC.

For my whole career (almost 25 years now) I have loved automating tasks. Early in my career I really struggled with the balance of how much to devote to automation vs just trying to get the job done fast.

One thing that helped me was to invest in building simple libraries and tools that I would commonly use to make the automation faster. One such one was a script (called super-script) that would build the skeleton of another script, prompting for all the command-line parameters that I needed for the script and including all the command-line code that would convert the parameters into an easy to access data structure. This script would also prompt for help text for the params and create code to print that out when `--help` was fed into the output script.

This did two things:

  1. made the new script easier to write

  2. made the new script *MUCH* easier to share

By getting #2, the payback for automation was way way faster. It might take a year for payback if only I used my script, but it might take only days if 30 or 40 of my colleagues also used it.

I do that to. My script is called “docker_genesis.sh” - can you figure out what it does?

If you haven't done it by hand at least once, you're going to fail badly at automating it.

Or "automating" it but wrong and/or with enough banana peels tucked away in the code to cause 60 hours of net frustration and refactoring effort, down the line.

But hey, it passes CI so who's gonna know?

If you do it by hand in 6 minutes, then you can play Factorio for 5 hours and 54 minutes!

This is my mindset as well: a sporadic six-minute task just needs good documentation, but a frequent six-minute task needs automation.

It's all about maximizing downtime.

Automation is not only about time saving. Humans just can't be trusted to perform boring, repeating tasks reliably. Familiarity can lead to all sorts of issues, especially with mental fatigue built up over time.

A computer does not have curse of knowledge, and is much less likely to change its behavior (forget to execute a line of code, or run a command it was not instructed to do) in the long run.

An unoriginal tweet from MARCH. c'mon

Fishing for followers on HN, I should consider this

* Looks around

"We down bad here too?"

I don't see any more than a tweet with the same text as the caption. What is this, reddit?

Also very confused, and that twitter account and instagram account seem really botty to me for some reason. Generic click baity fortune cookie posts. Says shes a dev but dont see any portfolio or linkedin or project shares. Instagram is just a bunch of click baity sexy photos that give me a little bit of a deep fake vibe. But what do I know, my feeble human brain cant discern between reality and computer marketing fever dreams anymore...

https://github.com/camilaleniss

Looks pretty legit...

Why use many words when few will do?

Sometimes the HN comments are the real article.

I've been using a custom-baked "cluster access" toolkit for automating things for Cassandra (and now other clusters). It takes a while for the playback to happen timewise, but it does start to accumulate, especially as the number of machines increases.

But it becomes a massive load of configuration, and I've had to rewrite from scratch three times.

When you have even just preprod and prod, it can be hard to get the savings, but once you have infrastructure copies in four or five different environments and five or six AWS regions, then things start to really pay off.

My "tool" tries to abstract the means of forming a set of infrastructure, the means for delivering commands to it (ssh, AWS ssm, kubectl, dockerrun, or other means), tries to abstract the clouds to do multicloud (but I've never tested this). Man is it hard to do all that. Config config config, context context context.

But it pays off eventually. Someday I might get the time to open source it, probably when the current job kicks me to the curb and no one has a use for it but me. I won't try to say it would be useful to others though.

That's the kicker on complex automation, with Conway's law in the mix, it is very hard to see what hidden dependcies/hardcodes there are based on your stack, software, team/company, policies, etc. To abstract that is such a mound of configuration it almost starts to collapse upon itself.

It took me three years before I could fully automate a cassandra cluster in multiple regions, data loads, spawn loadtests, run them, and tear it all down in near pushbutton. But man do you get more powerful when you can script stuff like this. Woe be to them that has to try to parse them apart.

I've taken to writing a sort of "workflow engine" that is based on filesystems and subdirectories, so I organize the code into individual scripts I can run together, or I can run it on a parent directory and go "run all these scripts as a workflow". So now I almost have an adhoc workflow engine, which is probably a "any sufficiently advanced code has a bad LISP engine in it" joke almost.

The issue is no one makes a general thing like this I can see. Ansible is tied up in ssh, saltstack relies on their UDP delivery. Most things like those two have bolt-on for k8s but it is hokey.

I do think the "COMMAND LINE" is the most universal standard of automation, but I think it's a bit underappreciated.

Automation is good for reliability, machines are much better at being machines than people are.

The problem is most of the "I automate everything" people seem to use handwritten bash scripts for everything, instead of purpose built tools for whatever it is, even when those tools exist.

6 hours is worth it if it entails a lengthy logic decision tree to assist us six-month later.

Automation tool is often shockingly more complex than initially thought.

Just think of all the features that Kubernetes do, they all used to be some ghetto scripts here and there. Another good example is Airflow.

https://xkcd.com/1205/

Caveat: it's worth spending more time automating a task than actually doing it if the task is tedious enough.

Other XKCD: https://xkcd.com/1319/

Thanks, saved me looking for them!

I was going to post this, but you beat me to it. This is a really useful chart!

"ACROSS 5 YEARS" !!!

Did she tell what she was trying to automate?

Reply on news.ycombinator.com