Rabbit Holes: The Secret to Technical Expertise | Hacker News Reader

Rabbit Holes: The Secret to Technical Expertise

blog.bityard.net

276 pointsdailymorn57 comments

The trick is knowing how deep to go, which depends a lot on your goals. If you just want to learn for your own curiosity, rabbit hole forever! If you want to build useful, applicable knowledge, there’a a peak of usefulness after which there are diminishing returns. So being self-aware enough to know when to pull the plug on your rabbit hole is pretty important (professionally).

Also certain kinds of rabbit holes are way more useful than others. If you’re rabbit holing into some obscure, arcane system, and the stuff you’re learning isn’t challenging you fundamentally, you’re probably wasting your time (I call these “rat holes”). If your rabbit hole will yield new fundamental insights into something important...now that’s a good rabbit hole!

The best depth before you hit the knee in the curve is undergraduate-level textbooks. Sometimes you have to go deeper, but that's the level that is presented in a form that's easy to grasp and can be quickly applied without using too arcane language and abstractions.

Also: Handbooks. For instance, the Art of Electronics. And I recently picked up a mechanical engineering handbook, and the breadth of the topics covered at a level I can quickly understand is very refreshing. (I'm a physicist, but I mostly do mechanical engineering and technician work, with a bit of aerospace and electrical engineering and material science.)

And combined with youtube and Wikipedia, you can learn a lot very quickly.

> The best depth before you hit the knee in the curve is undergraduate-level textbooks.

That matches my experience as well. Having graduated from two fields at university, the most foundational knowledge came in the first two years. The third one is a deepening of the first two but can be skipped if only basis are needed. Then master degree is just playing with higher level concepts and linguo.

>Art of Electronics

What would you suggest before The Art of Electronics?

Practical Electronics for Inventors https://www.goodreads.com/book/show/687619.Practical_Electro...

Title is slightly wanky, but it's the best non-textbook beginner guide I've found, since it actually does contain all the math & theory a hobbyist will need. The explanations are very clear and most motivated people with high-school math should be able to follow along. It's not the best to read back-to-back, but has detailed explanations of lots of areas of electronics that make great primers/refreshers. I had it out last week for a quick refresher on notch filters. Doesn't go into tremendous depth, but more beginner/hobbyist friendly than Art of Electronics shich is similar (and excellent) but aimed at EEs instead of hobbyists.

For something even more beginner-oriented any of Forrest Mims' books (formerly published by Radioshack/Tandy) are great but a little dated, and don't cover anything more modern than discrete logic and don't provide much in the way of theory. Excellent introduction to electronics books though, many fun projects for getting kids into the hobby who want to dive a little deeper into electronics vs. hooking up modules to an Arduino.

For Mechanical engineering, I strongly suggest picking up a used copy of the Machinists Handbook. Excellent handbook for MEs & Machinists that contains nearly any type of reference information you could want on those topics.

There are also Python holes. Holes that suck you in and don't let you go. People with an obsessive personality type have to be careful not to enter them because when you leave them it might be decades later and you are a husk of what you were.

yea, but in that case the problem isn't entering the hole, but the obsessive personality.

As an obsessive person myself, knowing how to stop, how to put things down and "grasp for air", is something I had to learn to do (meditation helped)

I've been writing software for a while now, and "how deep" is tough. In some sense it's just all rabbit holes and yak shaving.

At least for me, the promise of the pomodoro technique is the guarantee that you will take stock of the value of continuing to spelunk or shear, every 25 minutes.

As a newish manager this is something I find difficult to convey. If someone wants to go down a rabbit hole for a day or two and nothing is on fire, by all means. But you can’t do it every week and you can’t disappear for weeks at a time.

For a while I was trying to peddle a metaphor about deposits and withdrawals from a reputation bank, but it didn’t seem to land.

Edit: pedal -> peddle

On a thread about rabbit holes, it seems safe to suggest that “peddle” was the verb you were looking for.

That's a nice analogy. In a professional setting, spiking on research is time constraint always, and if you start to meandre too much, you will deplete a reputation. But on your own, and with clear goals, as someone here mentioned, by all means dig in as much as you desire if it fulfills you.

If by rabbit hole you mean taking the time to know how the system their responsible for maintaining, updating and troubleshooting in production works, I'd say give them as much time as they need. If you think that's taking too long it could be an indicator the system is either too big or too complicated.

I love taking my own time just as much as the next engineer, but if there are extrinsic factors calling for short-term solutions, one can’t take several weeks or months to “research the system” before the crisis is solved.

Shortcuts is my personal mortal enemy. I talk to people that "really want to learn topic X" but: a) don't want to read any books, b) don't want to watch any training videos (from nuggets in the old days, to youtube tutorials now). They just want a quick and dirty solution to everything, always. That comes with a price.

People want shortcuts. I don't like or use shortcuts 99% of the time.

My experience is that when you go looking for X to serve a need today, you discover Y and Z and guess what.. Y and Z will come very handy tomorrow, when in a call/meeting about ABC, miraculously Y and Z are similar and you speak up a d impress everyone and get the job/contract.

Knowledge that compounds in such a way becomes useful in the long term. The Y and Z mentioned may be tangential in a current situation, yet as a technology stack evolves they can become integral parts to more complete and better fitting solution.

I frequently get emails from people which follow a predictable pattern:

"I'm doing a PhD in .. I saw your blog post .. can you point me at a tutorial for advanced-subject?"

I often wonder why people don't do research, and what would happen if tutorials just disappeared. People would have to do their own research and learning. Maybe things would take longer, but I think the people would benefit from it.

I'm not sure if I can speak for everyone but sometimes I do this in the hopes that it could lead to a connection with the author or potentially deeper insights and discussions on the topic with someone more knowledgeable. Usually I go off and try to figure out what it was I was asking in the first place anyways

I'm happy to answer questions, and sometimes I can point to non-obvious resources. But mostly I feel that the approach would/should be different for trying to discuss things than "Please help me do my homework"

> what would happen if tutorials just disappeared. People would have to do their own research and learning. Maybe things would take longer, but I think the people would benefit from it.

I sincerely doubt it.

When the barriers to learning are raised, we should expect to see less learning, not more.

I learn a lot from quality blog posts and YouTube videos, on all sorts of topics. I have no interest in taking an undergraduate degree in aeronautical engineering or political science, but with easily-consumed 'tutorials', I can absorb a basic understanding of all sorts of topics.

Were these resources to go away, I would certainly be less knowledgeable, not more.

Also, I'm not sure that People would have to do their own research and learning means anything in the first place. We're talking about the quality/quantity/availability of learning materials. That's all.

Admittedly, I don't send personal emails, but on HN or Reddit, I'll often ask experts or people who are at least knee-deep in a topic themselves (or find previous instances where expert(s) was asked).

It's not about laziness, it's just that as somebody who doesn't know anything about established papers or books of a certain field or how it's structured, it can help a great deal to get started if somebody can point in a general direction of where to spend my attention (which is a limited resource). As somebody with dozens of interests across a bunch of disciplines, drawing on existing knowledge has been such an immense time saver and help that I'm always grateful for. There are a lot of books and papers out there. And not all of them are particularly good. If other people have experienced them and share that experience, why shouldn't I want to draw on that?

Also, I've gotten a response as often as I haven't. And when I don't, it really isn't a big deal. You don't need to respond to every request.

> You don't need to respond to every request.

True, but I feel a little bad if I don't at least say "Please write a tutorial yourself, people would be pleased".

Sometimes useful discussion occurs, but mostly it feels like "please send me the codes / do my homework for me". It's hard anyway to judge intent.

I'd love to know what percentage of the population has any interest in self-directed learning.

It may well be smaller than we hope. And HN - probably software in general - is unlikely to be representative of the population median.

The good thing about the Internet is that there is a lot of great information that was never available before.

The bad thing about the Internet is that it is interspersed with incorrect or misleading information. When tackling something outside your expertise, and outside the expertise of a large number of people, it can be hard to discern the two.

And then we are forced to go back to our web of trust. "I trust Steve Kemp is knowledgeable, and he can get me bootstrapped."

I honestly don't know a way around this outside some sort of curated expert-reviewed system.

Of course, being snarky, I might answer, "I'm unaware, sorry. However, after you learn, I'd love it if you can create such a tutorial to help others after you!"

Yeah sometimes the best replies I can make are "I'm sure if you wrote it people would be greatful". I don't like saying "no" to people, and often go out of my way to answer quite complicated questions.

But when receiving a brief "Point me at a tutorial for XXX/YY/ZZ" email it's hard to discern whether the person wants to learn, is incapable of using a search engine, or is really contacting the wrong person.

That's what leads to tech debt!

Reminds me of how, as a kid, I tried to solve Fermat’s Last Theorem. I didn’t have a chance in hell, of course, but my attempts enabled me to learn a ton of advanced math and even finding on my own some results that turned out to be already well known (e.g., Euler’s formula). It's one of the purest forms of learning but as an adult, I no longer have the time or freedom to do it to the same extent. Still, the basic approach is worth keeping. You just need to limit yourself a little so that you don't end up too deep in the rabbit hole, spending a ton of time on things that you really don't need to know.

I had a similar approach with the following equation:

What does the infinite series of 1/(k^2) converge to for k=1 to infinity.

When I was in high school, I spent weeks going down this rabbit hole of mathematics, failing to solve it. Years later I discovered that this equation is a special case of the Riemann Zeta Function, which was also first studied by Euler. His proof is really elegant. The funny thing is that with some math equations, sometimes you intuitively can see that there is something important or far reaching about an equation, even if you arent skilled enough in mathematics to discover it. Some kind of curiosity draws you in

Or positioned another way, this is Yak Shaving:

Any seemingly pointless activity which is actually necessary to solve a problem which solves a problem which, several levels of recursion later, solves the real problem you're working on.

http://yak-shaving.urbanup.com/1164318

The "right" solution to get a quiet login is to

   touch ~/.hushlogin

in your home directory.

It's documented in the login(1) man page.

There's also a global /etc/hushlogins file you can create to take care of all users.

Rabbit holes are why I grew to love Haskell and pure-functional programming in general. For better or worse, Haskell (Agda, Idris...) doesn't let you compromise very often.

You could argue that this is terrible for productivity, and you might be right, but at the same time, means that a lot of people have done a lot of research on how to do things the "right" way in Haskell, and any project I've done with the language has led me to learning a whole slew of new things about type theory, category theory, and PL theory in general, which I can then take and apply to "productive" languages.

This aligns precisely with a memorable interview I gave to a PhD student who wanted to do an internship at a startup.

"The thing I love about Haskell is that even the simplest things become research projects"

However I understand where you are coming from. There's something to be said for environments that encourage learning.

Every now and then I get mildly embarrassed because someone has managed to summarize my long-windedness to a beautiful and elegant sentence. The line "The thing I love about Haskell is that even the simplest things become research projects" has done that. I'm going to steal that.

This article describes pretty much the entire reason I got in to software development, why I'm still doing it, why I'm here today, why I run Linux on my main machine, etc.

I have an undergraduate degree in Physics for largely the same reason.

I'm increasingly convinced that this is just a mindset you have or you don't.

For me, the world is a place to tinker about in, learn, build stuff, break stuff, etc.

I'm currently tinkering about with my homelab setup, tweaking the security, setting up redundancy, etc. Why? 'cos I can, and when you hire me you get all of that.

It's certainly served me well so far!

As one who simply tinkers with Linux but is not a guru, I found this article quite inspiring! I had no idea just how deep that sort of rabbit hole can go. I feel inspired to try to delve behind the scenes for myself... The author has made my day :-)

Yeah this is just overkill. I haven’t edited /etc/motd in a while so this was news to me but this is just code creep and makes me sad.

Same here. My reaction at each step is “how do you delete this subsystem?” and “what poor person wrote this?”

Still, nice learning exercise for the author.

Exactly. I'm sure somebody wanted it at some point somewhere, but now it's on every system. No wonder people keep starting new minimal Linux distros.

That. They could’ve just supplied a default .profile / .bashrc. It’s these kinds of bells and whistles that causes most security bugs due to additional attack surface. Is it the root user fetching the news with wget/curl from a remote source?

All of systemd is the same shitfuckery. I move away from debian to slackware and voidlinux (when using BSD is not possible) just to avoid it.

This is the reason why people who don't respect coding as a discipline end up producing terrible code.

In my professional life I've run into a lot of people who will say "the idea is already there, it just needs to be coded up".

They will then come across common coding issues like keeping things in version control, adding tests, network config, and language specific things. They decide they don't have time for this particular rabbit hole, because of course they already know what they want.

The good people will over time end up investigating everything, even if it doesn't seem relevant to your LOB application. And they produce much better code because of it.

Its not that people don't respect it, its that they don't appreciate it. And the reason they don't appreciate it is that those things are invisible to the end user.

How much extra money would you pay for an electrician or a mechanic who does the equivalent of what you're suggesting with code? 1.5x? 2x? Most people I know always complain that their garage overcharges them or that they got ripped off by some contractor or the other...

As someone coding for my own business, I often find myself deep down a rabbit hole and have to get out and force myself to just use that ugly patch.

It's a balancing act, because it hurts me to just leave something without completing it "the right way" but I have found that if I try to do everything "the right way", I would get so little done.

I try to keep this in mind when reviewing or working on other's code. Some things will anger me at first glance but the problem is, they were probably just prioritizing different things at the time than I would have prioritized right now.

I love this kind of stuff, it's the most pure kind of learning, no end goal but satisfying your innate OCD. I've gone down these rabbit holes in tech and alternative fields - politics, philosophy, cooking. It's crazy, but it lead me to some life long hobbies.

I'm not sure about "technical expertise", as most of the stuff you learn while doing this is highly specific, but perhaps the real expertise is having the toolset and determination to do this at all. To keep taking screws out of the system and removing the covers until you find out where the system makes a particular decision. Fortunately in this case you can go to the source; if you can't, or as a first attempt, strace will usually tell you where a particular program gets its information from. Harder to use here as you'd need to setup the login process itself with strace and there's privilege boundary considerations.

One observation I would make is how many layers of "plugin architecture" have been added here. Probably the original SysV login program just printed /etc/motd. Now we have PAM, and pam_motd itself also runs a bunch of tiny plugins, some of which also refer across to systemd.

I should probably write up some of my own deep-dive debugging stories, if I can still remember enough detail. The "why does structured exception handling sometimes not work on Windows CE" was a particularly time consuming one which involved disassembling bits of Windows kernel.

There was drama about the MOTD two years ago.

Here is the summary from LWN, https://lwn.net/Articles/726902/

So, blur the hostname in the SSH command, login to machine, where the hostname is ip-172.31.48.7.. Two things are funny here: First, the failed attempt, and second, the idea that anyone cares what the IP of your raspberry pi is.. If anyone cares to know, mine has 192.168.30.12.

haha, perhaps there was some profanity that had to be blurred out

Rabbit holes and their implications aside...

As a software engineer, a part of me has died because of the hardcoded strings (in the pam_motd.c patch). Why aren't these defined as constants in the header file?

Hopefully there's a good reason...

The metaphor critique paragraph seems to be missing the fact that name comes from Alice in Wonderland, where the protagonist falls "through a rabbit hole into a fantasy world": https://en.wikipedia.org/wiki/Alice's_Adventures_in_Wonderla...

I.e. it makes complete sense in context.

Most of us have fell down these rabbit holes of learning before; the author does a great job of illustrating the concept, showing the solving of an "ultimate question" about how/why something works the way it does.

I don't know if it counts the same, but I once went on a weird journey to solve a particular coding problem that I was certain was already answered. I don't recall the details, but I do recall thinking to myself "This seems like a common problem; a quick google should bring up a solution..." - so, one google search later, and:

https://xkcd.com/979/

At that point, I vowed to solve the problem - again - and post the solution (of course the fact that I can't recall what the problem was negates me being able to show you what I mean - my apologies - sigh).

I quickly found that to solve the problem, I first had to learn how to code X, which in turn meant learning about Y and Z, and also coding A, B and C. I ended up going down these rabbit holes; by turns learning and coding partial solutions, by others finding example code I could adapt to my needs (not always in the language I was working with, either). I spent more than a few hours on this task, but I eventually did come up with a solution.

And as I recall, I did post that solution to the thread where I encountered XKCD-979 - so future people wouldn't be inconvenienced.

We use platforms and layers that are at the level of branches and leaves on trees, supported by abstracted away structures hiding complexity that make them simpler to use as a platform higher up the tree (or down).

A researching traveler can go down the branch, to the trunk with lots of time rings and layers, and then further down there are many root systems and rabbit holes.

All sorts of functionality is abstracted away in layers down there, all the way to machine code and binary and everything before the epoch 1/1/1970. Epic programming and technology pioneer legends created it, they took complexity and made a simple platform layer for others to build on.

> Unless you got into tech solely for the money (you monster), it's stuff like this that we nerds live for.

I take offense to that! :P

The rabbit holes of my past are half of why my paycheck is where it is. (The other half is interpersonal skills)

It seems like the difference between a complicated, difficult system and a simple, easy system is whether it changes over time.

I feel like a Linux kernel in a NetBSD userland might be the best sweet spot, OS-wise, but I haven't used NBSD in a long time.

> * Keen to learn Istio? It's included in the single-package MicroK8s.

But this is a rabbit hole! His rabbit hole is about removing a rabbit hole.

I'm going to go meta with my comment and say that this article is a fantastic object lesson as to why linux faces extremely low odds of ever being adapted as a widespread replacement for windows or OS X. How many people (outside of your coworkers) do you know who would or could follow a path similar to the one laid out in this article?

I'm speaking as someone who has dabbled with linux since 1998. At that time, and now, there is a huge, confused ball of conflict between linux distributions, various levels of customization to those distributions, and documentation available on the web.

Here's a quick, recent example from my own experience. I decided I wanted to learn more about Emacs. I work in Windows at work, so I downloaded portacle. Great. Works great. Then you start watching youtube videos on how to customize Emacs, but most youtube videos are using Emacs in Linux. So you then have to find where portacle puts all the stuff that is in a completely different place than all the Emacs documentation and videos say it is. Great. And I haven't found anyone who talks about the benefits/costs and gotchas about whether to edit the init.el file, or the .emacs file, but now we're getting into the weeds.

So after that frustrating rabbit hole, I think, "I'll just install a linux distro in a VM." Next thought: "Which distro should I get?" Search engines are no help when it comes to finding the "best" linux distro because all I can find is superficial opinion pieces on the matter.

So back when I worked at an ISP in the early 00s, they used Debian, so I download Debian.

Try to install emacs in Debian, and I start getting weird errors about some certain package not being current enough, and apparently someone isn't maintaining/updating the install script for Debian. That's three days lost getting to that conclusion.

So I wipe that VM and download Ubuntu, because it seems to be the most widely used distro, and seems like a good distro for "normies" like me. Install emacs, no problem. Can customize by following youtube videos, no problem. Enjoying emacs now.

All of this comes at a high price, however. I've been messing around with trying to use emacs for a few months now, on my side time at work and at home, and it's only been in the last couple of weeks that I've been able to use the package in a way in line with the available training materials.

Normies have no kind of patience for this kind of ridiculousness. Techies justify these "rabbit holes" as enthralling and fun, and yeah, it feels good to finally figure some of this kind of stuff out, but overall, it's a massive distraction from actually USING the tools, rather than just futzing around, trying to get the damn tools to work in the first place.

If your're on Windows 10 you could've used LSW (Linux Subsystem for Windows), on older Windows something like Cygwin. When I have to use Windows for work, Cygwin is amongs the first things I install (actually I found a Cygwin derivative called babun pretty good, unfortunately I think it's now no longer maintained). I use Vim rather than emacs, but emacs works well too.

Reply on news.ycombinator.com