Checklists are hard, but still a good thing
utcc.utoronto.ca
utcc.utoronto.ca
The problem is that to use them broadly, you also need to implement operational-level changes. So if I made a startup that helped build checklists, the problem would not just be selling the software (which everyone has to do), but also convincing executives that checklists (even moreso than to-do lists) are worth investing in. The friction a checklist brings, at least in non-ultra-risk-averse verticals, might not be worth it.
I'm nowhere near as obsessed with them as that may make me sound; by temperament I'm definitely a go-with-the-flow sort of guy... which is perhaps exactly why I make sure to use them in certain important places. Go-with-the-flow has its strengths but also its weaknesses. Selective augmentation with checklists means I can often end up with the best of both worlds between that and being "detail oriented" for important tasks.
I have been called "detail oriented" a few times in my professional career, and I sort of laugh to myself because it is absolutely and utterly untrue. I'm not detail oriented in the slightest. It's all offloaded to the computer, with my unit tests, checklists, and such. I just know I need that augmentation and am quite aggressive about getting it precisely because I know I need it.
too often i'll have a sudden thought about "oh, i need to make sure to fix X" or "I cant forget to pack Y before the trip". My options to deal with those sudden thoughts are:
1. Do it immediately, which isn't always possible
2. Do it later, which means that I'll obsessively stress out about it because "what if I forget to do it? I need to make sure this stays at the forefront of my mind at all times."
3. Put it on a checklist and give myself permission to stop thinking about it until its time to do it
My brain already went through the hard work of figuring out what to do, so I should cache that result in the todo list. later I can do a quick O(1) lookup of what do and use all of my focus on doing that.
I often have to travel for work. Sometimes I have a fair bit of notice in advance and sometimes I don't. In either case, checklists are super handy for me. I generally use either Apple Notes or Obsidian for these (previously Emacs org-mode but it's less convenient for the "add an item from your phone" case than the other options).
For the "notice in advance" case, I'll start a packing checklist pretty much the moment I know I have to travel and I'll immediately write down just a sentence or two about the purpose/objective of the trip and add any special things I might need (e.g. a Saleae logic analyzer for debugging or a GoPro for documenting). As things get discussed and the trip gets closer I'll keep adding things to the list.
For the "not much advanced notice" case, I'll look back at previous checklists to find one that has a similar objective and basically just copy it wholesale and tweak. Super super useful keeping those old checklists around, even if they're only 80% correct for the next trip.
ive never considered using emojis as the checkboxes to compare between not done, in process, done, or failed though.
A thousand times yes. Here's an older version of mine: https://honeypot.net/2023/01/13/my-travel-project.html
While I'm on vacation, I make notes about things I've forgotten. Near the bottom of that list is a reminder to add those notes to the template so I'll do them the next time I travel.
My wife thinks I'm nuts. OTOH, I'm so much more relaxed before we depart because I know that I have everything I'll need for a fun, comfortable trip. And if I don't, it's going to be some relatively minor long-tail thing that's never come up before, not something basic like "pack underwear".
While I follow this approach generally pre-travel, I find that on occasion I can end up taking the 'regulars' out of my travel kit for use and absentmindedly place them back in their 'normal' locations when using them as part of my regular routine.
When it’s packing time, I treat that travel kit part as its own separate list and validate that each item is still in there, or temporarily borrow it from my normal setup. It’s also one of the first things I unpack when I get home so that I can brush my teeth that night.
I go even a step farther and some unique items I store in the travel kit. Not sexy, but my manual (rotating) nose-hair trimmer is small and lives in that travel kit b/c at least it has a home for when I need it, and doubles that I'd have it "just in case" when traveling.
Small tweezers, a flat nail file, small nail clippers, little comb, tide pen, folding travel toothbrush, extra tray of razor blades (in a zip-loc bag), sliver of bar shampoo (also in that ziploc bag), some hair-care products that I've transferred into tiny plastic screwtop containers from Daiso (search for "5 gram cosmetic containers"), a refillable travel perfume atomizer, etc. etc. etc.
Cork Puller (two thin strips of metal that slide down the edges of a cork) since that usually doesn't draw the ire of TSA, little round hair brush that folds up and has a sewing/button repair kit built into it. Also: some minimal medicines! Tylenol, Benadryl (runny nose), couple blister packs of DayQuil/NyQuil, anything that'll hold me over until I can walk/taxi to a convenience store.
All that stuff is in a little pouch that I keep next to my socks (https://www.amazon.com/dp/B017SKRWL4 - bagsmart electronics travel organizer case). Grab 5 socks, 5 underwear, that pouch, into a packing cube and I'm 99% ready to go. Worst case: buy shirts and pants at your destination, but brush, shave, groom, bathe are all taken care of.
I also have job interview checklists, beginning / ending job checklists, repetitive admin checklists (beginning/end of month, beginning end of year, beginning/end of financial year etc...).
For all of these the checklist template is something I edit after one of these events after thinking "man, I wish I'd thought to do X" or "I wish Id thought of doing X a bit earlier than I did".
OMG. I showed my friends the wiki I made for a recent job hunt with details about every job, every person I talked to, a status page for “initial contact”/“recruiter screen”/“tech screw”/… etc companies, and a timeline of events. Some responded in horror. Others started taking notes. I’ve never been a PM, but I PMed the “get a job” to the extreme.
Then a post-review for things not used, things missed, etc.
Also kinda related, but I'm a huge fan of the "getting things done"[0] philosophy. Putting things down on a list, helps ease so much of the anxiety about not knowing, planning the next item, and most of all, relieving the cognitive burden of having to remember, which most humans aren't built for. There's a (personal and individual) limit to how many tokens our wet neural networks can hold, and I'd rather use them to focus on the task, rather than having to remember the task from memory.
Get that on repeat mode, and boom, you have checklists!
My wife and friends are at times annoyed by how anal I am about pulling out my phone to note things down whenever a new thought or task appears for me, but I wouldn't change anything about that. So much so, my task manager app is the only one app on my phone and computer that I have a subscription for (other than news and entertainment stuff).
Run the list manually, then automated each check point by point.
Voilà!
I've quit proposing them for this exact reason. Sometimes I just make my own.
Typically we have a non-traveling employee or manager do the final once-over and attestation, though they don't review the work necessarily. (Usually they just spot check the gear being shipped/sent to the airport, Pelican cases, Airtag trackers, etc.) This has been helpful for organizational purposes.
We don't even computerize them in any way outside of taking a cell phone picture of the final checklist and uploading it to a shared Slack channel for archival purposes.
It has prevented so many problems. It also fails to prevent some... which get added to the next checklist.
You probably don't want a lot of process reliability in anti-fragile contexts, like the startups you mentioned. You insist on reliability in fragile contexts.
Nassim Taleb (who coined the term), literally uses the aviation industry as a prime example of antifragility in his book[1], so I think our wires are getting crossed somewhere.
[1] https://www.amazon.com/Antifragile-Things-That-Disorder-Ince...
Aviation is absolutely robust though.
Do you happen to remember how Taleb framed aviation as anti-fragile instead of just robust/not-fragile? As an industry I suppose we're quite good at taking lessons-learned from failures and incorporating those lessons back into making aviation safer as a whole.
I'd argue that most Western government institutions (say, governments that have been around for 200+ years) are what you'd call "robust"; likely churches (the Catholic Church, since it's quite centralized), as well.
"Fragile" systems are systems like the Prisoner's Dilemma, the Tragedy of the Commons, tenuous alliances, societies/industries that are constantly in turmoil, etc.
Each flight and aviation as a market are very fragile, what is the entire reason its safety was designed to be anti-fragile. An anti-fragile safety process makes aviation robust, despite its inherent fragility.
A todo list is typically a set of unrelated things to be performed once and annotated as "done." Something may be performed periodically, like iOS reminders adds take out the garbage on a weekly basis, but each performance is really its own item. Sometimes a todo list will have parent-child tasks.
A checklist is a two lists: one a template and the other the record of performing the checklist. The template and performance instance records are a set of related things which may or may not be ordered or conditional. A special type of checklist is an inspection checklist which may include results from performing a particular checklist item as well as the fact that it was performed.
Checklists, especially in system operations and safety contexts, also tend to have a bit of a procedural characteristic, possibly including conditional and even looping constructs or sub-procedures.
- If HPA is energized
- [ ] De-energize HPA
- [ ] Enter radome
It's just a lot stricter operationally, which is why I think you'd get a lot of pushback trying to implement one for non-critical business tasks.
A checklist is used repeatedly, and it’s about completing a task (or set of tasks) to a certain standard. If you’re following a checklist, you’re aiming for consistency.
Having dozen of checklists that work for specific scenarios would be worth it.
I take the list for a completed trip and use it to plan the next trip of a like nature. It has been very helpful.
Two days into a trip our neighbor contacted us about our garage being open. We asked him to close it.
When we got back I found the automatic opener disconnected from the then-closed door. I asked the neighbor about it. He said the door would almost close using the automatic opener, but would stop and reverse. He disconnected the door from the opener and pulled it closed. (Certainly not the most secure thing, since anybody could walk up and push the door open, but arguably better than sitting open.)
I found the track was slightly bent on one side, as a result of banging into it with a heavy object while getting our camping gear out. It was causing the door to bind enough for the automatic opener to be triggered to reverse (presumably the "don't crush a human" safety feature).
When we left I pushed the button on the remote, saw the door start to go down, and left. By the time we were at the end of the street, with the door out of view, it had no doubt reversed and opened.
When I leave now I make sure I see the door fully close.
Gawande is a crazy smart surgeon and writer. The book is pleasantly short and makes its points really well.
I used to think checklists were used by reading the item, then doing the thing. I literally thought of them as a recipe that you would follow. Complete a step, check the box, repeat... This is typically referred to as a "read, do" checklist. In aviation this style of checklist is typically reserved for non-normal operations—procedures that you wouldn't use often enough to commit to memory.
The other style of checklist is "do, confirm". In this style you complete a procedure by memory, and read through a checklist to ensure you didn't miss anything (no box ticking, you just read the items and confirm to yourself that they're complete). In aviation this is the style used for normal operations, and for the initial action-items in an emergency (which although not commonly used, must be committed to memory because they are time-critical).
Because you're expecting that the procedure is completed by memory, a "do, confirm" checklist can be extremely brief. You do not need to write in detail what each step involves, you just need a word or two to name the step. Additionally, they're an extremely low operational burden; it takes a couple of seconds to read through a "do, confirm" checklist but the upside of catching common errors is significant.
A checklist is an established, recurring procedure that is ideally memorised, but despite that one goes through each item on the checklist to make sure nothing is forgotten or performed out of sequence or twice.
A to do list is an ad hoc collection of tasks that need to be performed. The list is built for each occasion rather than covering a standard procedure.
It's really a good podcast (pretty much all of his podcasts are)
https://timharford.com/2023/07/cautionary-tales-a-fascinatio...
I finally had some time to spend on it, and here's the result: https://checkoff.ai/
Would be grateful for any feedback!
Ever go try to cook something new and you read "Pre-heat the pan on low-medium...", and your programmer brain just can't take it? What kind of pan, what's low-medium on this burner, how much pre-heating are you talking about? These can't be all the instructions.
And perhaps like programming, it takes a few recipes and a few burnt steaks for you "not to worry" about that, you know what's good enough eventually. These lists (and algorithms) are never completely thorough.
None of these descriptions is perfect, but each is less likely to result in a burnt steak.
There is one time in my life that I recall legit burning a steak. I did what I had countless times before. Heated the pan until the oil started smoking, put on steak, and reduced stove temperature. Just like how I would have written the recipe before without a second thought. This time, however, the outside was thoroughly burnt before the inside even started to cook. The difference was I was using a cast iron pan for the first time, which has a lot more thermal mass than what I was used to. My old process relief on the steak cooling down the pan.
For recipes I'm reading, I've almost always found the temperature and time details to be nearly useless. If the recipe says to make at 400 F for 30 minutes, I bake on "high" (450F) until done. If I'm in someone else's kitchen, my cooking turns out a bit worse than when I'm at home.
This is a problem you always run into when writing down a process. You need to rely on the knowledge of the person following the process to apply it correctly to their specific situation. Trying to prescribe every detail does not work well.
"Low-medium" is just bad instructions. The recipe should be more detailed.
Anyway, what you are complaining about on your example is just jargon ignorance. You need to learn some stuff before you understand recipes. That's not really what makes programming hard. But it does make learning new things hard.
It's not programming experience that makes you ask those questions but because they are basic cooking questions you would ask.
Checklists are either checklists or a set of instructions to follow.
Good instructions would specify some of those attributes, other times experience helps to understand what general pan you might use or what medium-low heat looks like. Exactly to your point, it takes a few burnt steaks to figure it out.
Same reason we haven't typed "cc" on the command line to call the C compiler on individual files for about 30 years or more.
I mean, I've surely compiled orders of magnitude more C files without typing "cc" on the command line over the last week. But it's actually pretty common for me to tweak options like -mcpu, -m32, -pg, -Os, or -std=c11 -pedantic (not to mention diet cc) by running a "cc" command line directly.
Similarly, I often run Python or JS code in the REPL or in Jupyter rather than putting it in a file. The rapid feedback sometimes helps me learn things faster. (Other times it's an attractive nuisance.)
But I may be a bit of an odd duck. I've designed my own CPU, on paper. I write assembly code for fun. I've implemented several different programming languages for fun. I like to know what's underneath, behind the surface appearances of things. And that requires experimenting with it.
I still remember years ago trying to convince one dev to use make on a package with 20-30 source files.
https://gitlab.com/kragen/bubbleos/-/blob/master/yeso/admu-s... is the entry point to a terminal emulator I wrote, for example. `make -j 8` can build it with GCC from a `make clean` state in 380ms, but if I, for example, `touch admu-shell.c` after a build and run `make -j 8` to run an incremental build, it recompiles and relinks just that one file, which takes 200–250ms. So the incrementality of the build is saving me 230ms–280ms in that case.
Without -j, a nonincremental `make admu-shell` takes about 1100ms.
But if I instead run
time cc -Wall -Wno-cpp -g -Os -I. -std=gnu99 \
admu-shell.c admu.c yeso-xlib.c yeso-pic.c \
png.c jpeg.c ppmp6-read.c readfont.c ypathsea.c \
-lX11 -lXext -lpng -ljpeg -lm -lbsd -lz \
-o admu-shell
That's something, to be sure; it's a saving† that makes the compilation feel immediate. But it's really pretty minor. 900ms is small enough that it only affects my development experience slightly. If I were to run the build in the background as I was editing, I wouldn't be able to tell if it were incremental or from-scratch.
Unless it screwed up, that is, for example because I didn't bother to set up makedepends, so if I edit a header file or upgrade a system library I might have to do a scratch build anyway. The `make` incremental-build savings doesn't come without a cost, so we have to question whether that cost is worth the benefit. (In this case I think it's worthwhile to use separate source files and `make` for other reasons: most of that source code is used in multiple Yeso programs, and `make -j` also makes a full build from scratch four or five times faster.)
If we extrapolate that 700ms saving backward to 25 years ago when our computers ran 500 million instructions per second instead of 30 billion, it's something like 45 seconds, which is enough of a wait to be distracting and maybe make me lose my train of thought. And 5 years further back, it would have taken several minutes. So `make` was an obvious win even for small projects like this at the time, and an absolute necessity for larger ones.
At the time, I was the build engineer on a largish C++ project which in practice took me a week to build, because the build system was kind of broken, and I had to poke at it to fix the problems whenever something got miscompiled. The compiler and linker were writing their output files to an NFS server over shared 10-megabit Ethernet.
As another data point, I just rebuilt the tcl8.6-8.6.13+dfsg Debian package. It took 1m24.514s. Recompiling just generic/tclIO.c (5314 SLOC) takes 1.7 seconds. So not doing a full rebuild of the Tcl library can save you a minute and a half, but 25 years ago (when Tcl 8 already existed) that would have been an hour and a half. If it's the late afternoon, you might as well go home for the day, or swordfight somebody in the hallway or something.
So incremental builds at the time were totally essential. Now they're a dispensable optimization that isn't always worth it.
______
† 1200 lines of C per second is pretty slow, so probably almost all of that is repeatedly lexing the system header files. I'm guessing that if I took the time to do a "unity build" by concatenating all the C files and consolidating the #includes, I could get that time down to basically the same as the incremental build.
Start by automating the low hanging fruit.
After a few iterations you have a somewhat automated process and some steps that are harder to automate. Those hard-to-automate steps can be annotated with detailed instructions that expose opportunities to partly automate them. You can break down each step more over time. As you run the checklist, you’ll learn & iterate. Then with the newly broken down steps, you can automate what’s become automatable. Repeat forever!
[0] https://blog.danslimmon.com/2019/07/15/do-nothing-scripting-...
But any checklist needs to be a living document that has is easily updated and under version control.
Personally, I used MyLifeOrganized documents stored in Subversion.
I write them in (emacs) org mode in a way that I can “execute” them repeatedly. Have written about it: https://www.naiquev.in/recurring-checklists-using-org-mode-i...
I am mostly thinking about packing list, as it is common to overpack when travelling, so it can be good to have a list of things you think will be useful but are not. For example, if you are travelling in hotels (as opposed to hitchhiking ;)), you probably don't need a towel, because they are usually provided, so, to the do-not-pack list.
The problem with checklist, especially the ones that change over time is that they sometimes go out of control with useless items, a do-not-check list means "I removed this from the checklist, there is a good reason, don't put it back"
Also, critical business processes are great expressed as checklists.
He wrote The Checklist Manifesto - How to Get Things Right, has been interviewed a gazillion times about the need for the surgical checklist, and talks extensively about the challenges associated with getting hospitals and medical practices to adopt the surgical checklist.
NPR's Atul Gawande's Checklist For Surgery Success [1] is a short version of the problem, challenges, inspiration, and solution.
And more recently, look forward to Captain Steeeve [2] talking about checklists pilots use.
[1] https://www.npr.org/2010/01/05/122226184/atul-gawandes-check... [2] https://www.youtube.com/@CaptainSteeeve
The cost of reversing a decision is cheaper in software, than to say flying or doing hardware. Plus the spec requirements for flying and hardware design rarely changes as often as software.
Still swear by checklists and SOPs but have to be at peace that others don't see it the same way.
Of course, these checklists manifested as Excel spreadsheets.
IMO the best way is to start small, and every time checklist didn't catch an issue either modify existing item(s) or add new item(s). Organic complexity is the best complexity.
I keep several checklists - some I use several times a week, others every few months or so. If I notice something needs to be added to the checklist or removed, I do so.
It's always better to start with an imperfect checklist vs not having any checklist at all. With no checklist, you start from scratch every single time. Not starting from scratch allows you to focus on marginal improvements with each use.
"plans are useless, planning is essential."