Ask HN: How do you deal with Rabbit Hole Syndrome? | Hacker News Reader

Ask HN: How do you deal with Rabbit Hole Syndrome?

140 pointsphoton_off82 comments

I find myself striving for ideal and perfect solutions in parts of my work that might not matter much. Sometimes it's probably worth the time and detail, but admittedly, a lot of the time it isn't. It's just more fun and interesting to be "thorough."

This happens when working on something new and unfamiliar, often side projects. In the quest for a "perfect" solution, entire branches of mathematics or programming start to beckon me, and away I go... down a rabbit hole of Wikipedia articles, Stackoverflow questions, and Github projects. Even if I do find an adequate solution, it's not good enough until I can understand the class of problem and derive it for myself.

This also happens based on principle. When I encounter a problem that is actually solvable -- though it may not matter much, then I must conquer it entirely. As an example, for me it is unacceptable that something rated 5.0 with 1 review is ranked higher than a 4.9 with 10 reviews -- and I need to find the solution to this problem, even though it might only be a minor feature and won't matter if there are no reviews to begin with.

I call this "conditional" Rabbit Hole Syndrome. It also sounds an awful lot like the classic interview answer: "My biggest flaw is that I work too hard." Well it's true, damn it. These types of trivial things can easily become my "top idea" for days at time.

I am now able to admit it: I spend time working on things that probably won't matter that much, and I enjoy it. But how do I fix this? How do I learn to leave well enough alone? How do I learn the distinction between frivolous research and actually getting shit done?

A couple of edits: After more thought, this is really only an issue with work on my side projects. I also think it's due to lack of accountability -- I report to myself. As such, I tend to work on "most interesting first" and avoid schlep. So I'm wondering: how do you motivate yourself to schlep?

View on HN

If I had one piece of advice it would be: Descend the rabbit holes!

Rabbit Hole Syndrome is a symptom of having a curious, persistent, independent mind. If there were a drug that cured Rabbit Hole Syndrome, and everyone just worked on Shipping Their Products and Not Asking Questions, there'd be no one left to make the interesting discoveries that make the world better in the long run.

There are very famous quotes from Richard Hamming (see "You and Your Research") and Richard Feynman (see "Surely You're Joking!") about the importance of working on problems that seem trivial at first. Not only do they help you enjoy problem-solving for its own sake, but if you work on enough silly problems, then eventually the odds are good that you'll stumble upon something that other people will later think is really important. Incidentally if you read Thomas Kuhn you'll find that this tends to be how scientific revolutions happen: someone is bothered by some tiny little thing that doesn't quite make sense according to the prevailing theory, starts digging at the little cracks, and finally the whole system comes crashing down.

The major benefit of going down rabbit holes is that is opens yourself to serendipity: sometimes you'll turn out to be right about something for the wrong reason, or discover something that you later realize contains a solution to a seemingly unrelated problem. The more rabbit holes you've gone down, the more they start to connect.

In addition, frivolous research helps you develop a very "bottom-up" view of the world. If you know the details cold, then you are better able to see through high-level BS ("conventional wisdom") and evaluate things on their own terms. You'll find that regardless of what's optimal, most things are done a certain way only because they've always been done that way, and that regardless of what's true, most people believe things only because other people believe them.

Of course there are costs to all this -- in particular, "schleppers" will resent you for being irresponsible, and most people will think you're crazy if you ever explain what stupid little you've been working on lately. Which is why Rabbit Holers should try to should avoid actual responsibilities to the greatest possible extent and absolutely not care about what most people think about their work.

Anyway, to sum up, if you're lucky enough to be in a position to descend rabbit holes without impoverishing your family or bankrupting your company, I say go for it. It strengthens your most valuable asset (your mind), and who knows, maybe one day you'll discover something you can teach the rest of us.

big round of applause. you nailed it completely

Yet the conflict between "It's the trip, not the destination" and the fact that your salary arrives at the end of the month can cause much grief when not balanced in the mid term.

I second this. The gift you have is the drive and passion to learn by your self and explore. Let it go free and you'll become an expert. Soon or later you'll be able to turn this knowledge to profit for you and the collectivity.

Our education system is broken in that we don't recognize and develop the skill you have. People are somehow forced to do things they don't feel like being doing.

My impression is that it's ADD. But It's sad ADD is called a disorder, because these people have the remarkable ability to be extremely efficient and concentrated in what they like and are interested into, well above the average. These people could be experts and genius in their field, provided they could develop their skill.

I agree with your advice to "Descend the rabbit holes!" but feel it needs a little more context. The real question here is not 'should I descend rabbit holes?' but rather how often or when.

If you are building something, you inevitably spend some part of your time writing code that is familiar and boring (let's call this WORK), and part of your time exploring unfamiliar code and concepts (RABBIT HOLES).

Going down rabbit holes is not work, it is a form of learning - in fact, in the world of software development it is the best form of learning because you have a highly relevant problem and supreme motivation to solve it.

At the end of the day, however, going down a rabbit hole is no different than reading a book, or articles on HN. It is not actually work, and fooling yourself otherwise leads to the blurring of what is work and what is learning.

You should always be learning, as much as you can. But you must be honest with yourself, there is probably more work to be done, and you can't possibly get on with it until you pull yourself out of the rabbit hole.

You're driving along a road and you notice a pothole. You pull over to the shoulder, put your hazards on, open up the trunk, take out a reflective vest and tape measure, then you begin to analyze the pothole. You spend an hour analyzing the depth and formation of the pothole and determine that the cause is due mostly to poor mixing of asphalt. You call the town hall and learn that the road construction crew uses a Caterpillar BG500E wheeled asphalt paver. After some extensive research you determine that poor mixing occurs from the inferior design of the BG500E's auger and that upgrading to the BG600D with its improved auger would cause better asphalt mixing and produce paved road conditions less conducive to potholes.

By now it's 9:30pm. It's dark and cold. You realize what your original purpose was: Dinner with friends. That was two hours ago. You missed dinner, but hey, you got some satisfaction.

The above was more of an analogy about Yak Shaving than Rabbit Hole Syndrome, but there are parallels. Like you, I used to be obsessive about details and solving subproblems. I used to come home from work and work on my own side projects for similar reasons as your own.

But then an advisor said something like, "You need to focus. If you want this thing of yours to succeed, you have to focus on making it succeed. Nothing else should matter." So I stopped my side projects and I became so effective at building our product that my employees wondered if I ever slept.

Stay on target. Make it to dinner.

Even within the context your product, surely there are cases where you're presented with a problem that can be solved with several different depths of thoroughness. How do you, generally, choose a solution which optimizes for, success, time, and personal gratification?

The first step is admitting that you can't optimize for every direction. Facebook has a motto, "Done is better than perfect," which I used to think was cheesy, but realized is quite brilliant. "Perfect" leads to satisfaction, but "done" leads to revenue.

Say you want to try alternative payment processing methods on your site. Your site it tightly integrated with PayPal, and you want to add Stripe. You could (A) spend 1 day to bolt on Stripe with a hack in the HTML, (B) spend 4 days and add enough views to the frontend to make Strip work in a non-disgusting manner, or (C) spend two weeks and refactor the entire backend with a generic interface that will handle an infinite number of payment solutions.

All three lead to success, but you can't optimize for both time and perfection. Set a time budget. Leave breadcrumbs for the next programmer. Instill a culture that says, "Our code isn't perfect, we know this, but we do what we can with what we have." Done is better than perfect.

Although I agree with your idea in principle, there are only so many times you can just hack something on top before you start to wonder if the refactoring technique is actually the better way to go. You could just end up with a hacky system held together with sticky tape.

With forethought, you should have built your system originally with method C in mind. If he had done this at the beginning instead of just hacking it together then all this work after wouldn't have been needed!

This is why you need to do two things:

1) Apply conscious planning and forethought. Some things are clearly going to be used a lot, some things might not. Use bolt-on hacks where appropriate, invest time for things you expect to be using forever.

2) When the longevity of the tool is not easy to pin down, test it out with a hack. Spend 1 day adding Stripe the hacky way, and keep a slushbucket of time budgeted for revisiting experimental hacks that look like they will last a while.

That's my general approach, anyway. Most of the time, a hack is good enough and anything more is a waste of time. The story is admittedly a little different for me though; for me, software is a tool to do my job- software is not my job.

I think I see it differently as we have just spent 3 years developing a CMS at our company and every decision has been a well thought out and concious one. We spent 3 weeks planning when we first conceived the plan.

We have only had to do 2 refactors in that time on 2 controls. That was more about understanding our customers needs more as we progressed rather than not enough planning though.

Previous to that I built a custom ecommerce site for a wholesale distributor. That whole process was more suited to a quick hack over a well thought out decision. That's why I say I agree in principle, so long as the project allows it.

Don't get suckered into premature optimization. If you're still small, bolting stripe on lets you accept money and get back to building the rest of the product. Sometime in the future it will make perfect sense to redo the entire payment system. Today is not that day.

You are right, I think I am being closed minded in how we plan to build software instead of revenue.

That's how I develop now to be fair and not how I would build a product if I were to build it for myself.

"Done is better than perfect" is brilliant. It is motivational, succinct, and easy to believe. From my particularly context of a side project with no real deadlines, however, I can have perfect if I want to. There's no race against the clock, nobody to report to, nobody relying on my results -- just me and my idea.

At any rate, I'll tweak the "done-perfect" lever more towards done.

Whether you think there is a race against the clock or not, time passes regardless; your mind changes, your body ages, technologies change, people change. By the time you're done with your "perfect" creation, it will be too late to matter.

"Perfect is the enemy of good" perhaps could (should?) be rephrased as "Perfect is the enemy of done."

Until the last few years, I think I was of the mentality that would follow (C) or (B), but have recently switched to (A) -- just ship it and fix whatever's not perfect later; the opportunity cost of not shipping is too high.

Facebook's motto reminds me of something from a Holman slidedeck: "Nothing great was ever not shipped."

Also, there is no such thing as perfect.

Always keep the end game in mind and put in place a process where you are consistently asking yourself how what you are doing is getting you there. If you decide to go down the rabbit hole anyway, you have to admit to yourself that you are procrastinating rather than solving problems. This works for me, and I am 2-3x more productive. As soon as I get distracted, like right now, I know that I am wasting time writing a post on HN rather than attacking my goal.

I have a simple heuristic: which solution is going to get me home in time for dinner with my significant other?

Gauge the impact the diversion will have to your core competency. If the actual benefits and time estimates are overwhelmingly positive, do it. If it's anything less, or is just to address a nearly non-existant bullshit edge-case, make note of the thought and move on.

Don't sabotage the larger vision for short-term gratification.

Heh. Are you Neal Stephenson? :-)

The sad thing is, this example, while ludicrous, is perfectly justifiable, at certain scales. If that town is, say a big one on the East coast, your two hours (or even half a day, or whole day!) are little loss compared to the savings that will occur from the analysis. Hell, your hypothetical "curious person" would be well worth the investment of a six figure salary for the city!

That being said, I guess it depends on what results these random maunderings produce, and it's probably a crapshoot as to whether they are more often than not to result in favorable outcomes. Also, I like your example (obviously :)

Unless it's a hobby or an intellectual exercise, a brilliant solution without a customer/market is a waste of time at all scales, destined to moulder.

Chances are, you'd approach Town Hall with the solution to all their pothole problems and they would ignore it. To make it worthwhile, you'd either have to have earlier signed a contract with Town Hall to figure out their potholes (ie. have a customer) or develop a business based on your solution (ie. develop a market).

To be honest I think that "it's not good enough until I can understand the class of problem and derive it for myself" kind of attitude is actually very positive. The problem with this attitude is if you do a "depth-first" kind of journey instead of refining the solution with successive passes.

So I suggest you to try to build the minimum viable working program ASAP. Along the way, write all the things you would like to improve in something like an Evernote note, just a few lines for every thing you want to address and make better.

Then if you have something working ship it ASAP, don't care about what other people will say about the sub-optimal parts of your work: many programmers trying to achieve perfection actually end with a mess of complexity that does not serve very well the purpose of the software, so there is little to be embarrassed for a programmer for shipping simple software.

In the second pass, refine every part with the same approach: find a solution that within the timeframe you have is better compared to the previous one, but will make you able to ship a new version.

Also when you face a problem, other than reading the existing literature, papers, and the proper way to do it, check if there is an intuitive solution that is comparable as a result (even if maybe not provable or not perfectly optimal) but much simpler to implement.

But IMHO the golden rule is: don't freaking care, ever, about what other people think about your work. Often perfectionism is just a form of insecurity.

Depth-first vs. breadth first, brilliant!

I get it now: If either one is taken to the finish line, they are the same (all nodes visited). But when left mid-way (time constraint), breadth first still is something workable while depth-first has a few components perfectly done while the others not at all.

>> Often perfectionism is just a form of insecurity.

Possibly brilliant too. How do I tell whether it is coming from insecurity or from my drive to not avoid shlep?

Often perfectionism in one area is a means to avoid shlep in another.

For instance I prefer developing software over marketing my product ( http://gridspy.co.nz ). This means that perfecting our product's software is the comfortable and easy option when what I need to be doing is picking up the phone and calling people.

Just be careful about scope when you say "refine every part with the same approach" or you'll end up with TextMate 2.

Here are a couple of random suggestions:

1) Try the pomodoro technique, or some other form of time tracking. When you're about to take a side path which may or may not be a distraction, you can decide how long it'd be worth investing in it. Once the timer dings, you can stop and evaluate if it was worth it.

2) A few years ago I started repeating in my head the phrase "real artists ship". Embrace imperfect or partially finished solutions that are viable.

3) Keep a list of things that you'd like to investigate more. I've found that the act of writing down the idea lets me stop obsessing over it. Later when you revisit the list you'll be able to cross off the things that you thought were important, but turned out not to be. By delaying work on these items, you're able to better explore the most important parts of the problem you're solving, and so your future self will be in a better position to evaluate which areas need deep research.

I think you're not being ambitious enough in your side projects.

I'm like you. I write fairly vanilla production code, then I go home and work on alien technology. And I used to have the same problem as you: I'd get sidetracked and sidetracked, and somehow I went from writing a fart app to reading about type theory.

So I started skipping the fart app part, and started learning some more abstract theory. The nice thing abstract theory is that it's abstract---it's not incidentally connected to anything, so there are fewer places for you to get sidetracked.

So go sign up for a math course on Coursera, or learn the lambda calculus. They are so alien from your everyday programming experience that you won't have anything to connect them to---until you do. But then you'll be coming at the new topic in the direction of abstract to concrete ("a trivial application of x") rather than concrete to abstract ("there's a greater truth here and I MUST understand it!").

That was my thought. Anything that can be grasped solely by reading wikipedia or blogs isn't actually very cutting-edge. Not to say that stuff on wiki or blogs can't be very difficult to grasp, if previously unfamiliar.

actually they can be very helpful - because you can be looking at old information through a new lens

This is an experience problem. As you become more experienced, and see how you've wasted your time coming up with a "perfect" solution that ended up not mattering, then you'll be fine not having a perfect solution.

At some point you realize your best days are the days when you delete more lines of code than you write.

But that being said, chasing rabbits is what will eventually make you more skilled than your peers. Which is awesome, except now you'll have put yourself in a perpetual category of being paid less than your worth because you don't fit the same performance evaluation criteria as everyone else.

All anyone cares about is a) are you easy to work with, and do you b) get things done to contribute to profitability.

Continutally stressing yourself out by spending more time on problems than they deserve and eating away at your work/life balance does not contribute to a) or b).

> All anyone cares about is a) are you easy to work with, and do you b) get things done to contribute to profitability.

This gets to the crux of the matter. I think we all here have an intuitive sense that being a bit of a perfectionist and regularly chasing a few rabbits down holes can lead to better long-term results. But it's very hard to quantify, certainly it's impossible to quantify objectively; all you can do is be a good salesman about it.

There is another path though—put your money where your mouth is. Start your own business based on your chosen principles and standards of software engineering. It will certainly bring you joy and if your instincts are right then it could even become a competitive advantage as your competitors drown under mountains of technical debt while you nip it in the bud at the first sign of trouble.

Yea experience of wasting your life in rabbit holes, combined with getting older while getting little accomplished may help you focus. Unless you really find something in the rabbit holes, then down you go.

If you never find anything and still can't help yourself it's some lethargy/depression ADD brain dysfunction. Make sure to exercise enough to be in shape, eat, etc.

> I find myself striving for ideal and perfect solutions in parts of my work that might not matter much. Sometimes it's probably worth the time and detail, but admittedly, a lot of the time it isn't. It's just more fun and interesting to be "thorough."

This actually has a label: Maximizers vs Satisficers.

http://en.wikipedia.org/wiki/The_Paradox_of_Choice:_Why_More...

And a little something I wrote about it regarding programming languages:

http://journal.dedasys.com/2006/02/18/maximizers-satisficers...

I think the key is to "choose your battles", and be a maximizer where it really counts, and try and be more of a satisficer for the things that aren't so important.

I was just going to post something very similar. Glad to see it's already been done for me. :)

I'm in the middle of The Paradox of Choice right now, and while the maximizing/satisficing thing applies to many aspects of life—the book was certainly not written about programming—programming is yet another aspect of life where we're confronted with a huge array of choices.

This is especially true in side projects. In the office, there may be a standard set of technologies; you're likely working on an existing product; etc. Additionally, there are probably a number of constraints, including time and product requirements. In side projects, there are hardly any constraints, and every single aspect of your work involves choice. Just deciding what you want to create is one massive choice. Then there are the big questions of "what programming language/framework should I use?" and "what database(-ish) platform do I want to use?" There's a good chance since this is a side project, you'll want to try out something new and cool to build your knowledge and gain experience. Those alone can be huge decisions. Then there are questions of libraries (which library? or build it from scratch?), etc. Heaven forbid you want to start thinking about your choice of IDE (time to learn Vim?) and tabs vs. spaces and every other little decision you can get caught up on.

I'm at least as guilty as anyone else of these things. I spend too much time focusing on the best possible outcome and the best possible technologies among lists of hundreds and thousands—because I want the best and because I don't want to regret any failure—rather than simply settling for something that's good enough and moving on to actually get something done. The result is generally painful and unproductive.

You should definitely set standards for your work—don't produce crap—but try bringing the bar down from "ideal and perfect" to something more attainable like "excellent" or "very good" or even just "adequate." This will help you focus on your real goals and appreciate your own success.

Another useful term/link: http://en.wikipedia.org/wiki/Analysis_paralysis

As with others on this thread, I question the rightness of "curing" this tendency. But you can certainly improve the way you harness it.

Something that's helped me: make sure you have multiple rabbit holes available at any given time. i.e. several problems, any of which is interesting enough to tempt you. Then you can make reasoned decisions about which would be the most rewarding to work on, while still giving in to the temptation of rabbit holing.

Another benefit: maybe by solving one problem, you'll discover the other was totally irrelevant, or a special case of the other. (At least for me, "Turns out I didn't need this to be perfect" doesn't carry much weight, whereas "Turns out I didn't need this at all" is quite convincing.)

This happens to me often as well. Although, I am not certain I would describe it as something I need to fix. All my practical experience, going back 15+ years has formed this way.

As a recent example, I just spent a long time on a tiny plugin for CarrierWave (the Ruby/Rails file uploading gem). This took me through most of their source code (always good to read code, and I could probably PR on their project now), new Rails internals and techniques, and mocking and stubbing with RSpec (in the process, I turned up a bug with RSpec, which they promptly fixed; and I have a better understanding of mocking best practices and clean RSpec).

I do empathize with feeling I'm "not getting enough done/shipped" (if you feel that way?). To alleviate it, I try to cut corners and just get something out.

Time-boxing helps me do this -- "accomplish X in 1 hour". This doesn't happen too often, however. I know, like you, I prefer the learning itself; and, I view all my spent time as building experience and a critical mass to accomplish work faster in the future.

Also, I find pair programming helps immensely. I am sensitive to the other person's time, and thus naturally refrain from digression when pairing.

As for motivating myself to "schlep". I'm not sure what you mean -- menial tasks? I do those when I'm tired or having down time.

Finally, as in your case, I do this on personal projects and not on "work". That said, it can make it hard to get your startup and product launched - your own "work". Again, disciplined time-boxing helps. I have not mastered this, and find myself regularly looking for good time-boxing tools.

I write a ticket for someone else to take care of it. Even if someone else is only future me, it's enough for my mind to accept that it is taken care of and I can proceed with the really important stuff.

Most of these tickets get resolved as "doesn't matter / won't fix" two or three weeks later - but myself.

I agree with this. I do theoretical CS research, and whenever I find a general problem that I feel must have been solved by someone but seemingly hasn't, the only way I can get the problem off my head is phrase it as a StackExchange question (usually on cstheory) and ask it.

First, it forces me to write down clearly the question that's obsessing me, occasionally leading me to realize that doesn't actually make sense or isn't that interesting -- all the more so because it is published so my standards are higher that if I just wrote it in a text file of mine. Second, it gives me the impression that people will work on it now so I can stop thinking about it for a while (even if no one actually cares about the question). Third, in any case the question is documented so it won't be lost.

In some cases I get insightful answers (sometimes, brilliant answers) that save me the time to hunt down a solution in an unfamiliar field or to work it out by myself, other times I end up replying to my own question because I stumbled on the solution later. In any case, it totally defuses the urge to think compulsively about a problem.

Catalog your rabbit holes if possible, and then review them before working on them.

What I mean is, if you see a problem to solve, and you are able to keep working on your current task and to solve the problem later, then do so... just note the problem. This will immediately make you much more focused.

Then at the end of the day, review your list of rabbit holes and try and determine which ones are necessary for the current project, which ones would be educational / you want to do, and which ones can be discarded.

Basically rabbit holes are a problem because they are long and narrow and do not offer an overview of the entire grounds, so before jumping down a rabbit hole force yourself to survey the big picture and to see if you can step over it instead.

You apparently enjoy the interesting stuff, and love the fact that there is actual practical benefit - though initially very limited - from doing something that's theoretically interesting. Letting that chance slip would feel like a waste.

It might help to limit going down the rabbit hole too much, by researching what could be improved without actually doing it right then. Save it in a list for later, do the schlep, then when things get bigger chances are you'll actually need to take on some of the challenges on that list. And the more useful they become, the more satisfying it'll be.

Kind of sounds like procrastination to me. You should have a clear goal, and work towards it, allowing no or few of these side-distractions to take you off the path. A real life deadline helps immensively as well.

Yeah, I would suggest you make sure to read the top answer at www.quora.com/Life-Advice/How-do-I-get-over-my-bad-habit-of-procrastinating

I found that to be quite helpful in maximizing your happiness by getting done what you most care about in the long term.

What else do you enjoy? Do you derive satisfaction from solving problems for users? Maybe you're trying to perfect something that's less interesting to you than another problem on your plate.

If positive user feedback 'gets you off,' as it were, try releasing something that isn't as perfect as you'd like it to be and see if there's any measurable difference in sentiment. I bet there won't be. Do it a couple more times and hopefully it'll break the pattern.

For the other case, try time boxing yourself so you can get back to the more interesting challenge. Make that other thing 'perfect' if you won't.

Never happens to me. I'm always eager to finish, release and go on. Good enough for the users? Publish. Not so good? Publish. Some will bear it, the rest will swear it and I'll know what to fix.

I'd very much prefer to pick up an apple that isn't perfect than having no apple at all because of looking for the perfect one forever.

A couple possibilities. If you really enjoy doing these things for their own sake, you can treat your side projects as a hobby and stop beating yourself up about it. Spend all the time you want going down the rabbit hole, and enjoy that it provides a chance to think deeply, making something as close to perfection as possible, and be fully present in the experience of creating.

The other approach I'd suggest is at the complete opposite end of the spectrum. Start treating your side projects like a real business, and create some process that connects real incentives to them. There are harder and softer ways to approach this. You can quit your job and now you have the focus in your side projects that you need to ship to get paid. You could also keep your day job and set up some automatic funds transfers into a semi-liquid asset like a CD or 401k, such that you leave yourself enough to cover the basic expenses, but so that the marginal income from your side project becomes significant enough to be motivational. Revenue is a great feedback loop getting you to focus on what matters, and letting go of what really doesn't.

You could also bring in another person, perhaps as a business partner, or simply have one of your friends agree to call you up once a week and have you give them a report. The key here is to clarify and draw attention in your mind to the work that you're avoiding, and the outcome that you want. In this process, it's important to vividly color the positive outcome be emphasizing the real changes it will create in your life.

For example: "This week, instead of shipping my product which will enable me to quit working for other people, spend more time with my family, own the high-performance german cars I've always wanted, and visit 20+ countries a year, I instead spend 6.5 hours working on a sort algorithm, and 5.4 hours fiddling with the presentation styles on the country-list drop-down menu in my app."

I used all of the techniques above when I was starting my first company, and they made a big difference in helping me focus on what was important in getting to market, and not OCD-ing out on things that were, ultimately, minor details.

I think the key is to:

1. Set goals and fixate on them. Imagine what it will be like to have achieved the goal. Get excited about it and keep reminding yourself. When you're not making progress toward the goal, make yourself imagine the situation where you spend 6 months making unimportant things perfect and never achieve the goal. Imagine all the other goals you won't even be able to set because you're wasting your time.

2. Make other commitments. Make plans to meet a friend for drinks at 9pm. You only have 2 hours after work to get anything done – don't waste those 2 hours! If you can stay consistently busy, you'll notice quite quickly that not using your time effectively will lead you nowhere.

I've been very affected by this myself.

It happened the most while attempting to study research papers.

A lot of times I would be unable to finish reading what I had planned, because I had stumbled upon an interestingly-looking concept and then proceeded opening up another paper on that subject and so on.

And then, of course, having returned back to the original paper, I would have to re-start reading from beginning to freshen up my memory.

Terribly exhausting process, so I can perfectly understand your frustration.

The only solution I found was to un-clutter my computer 'work-space' as much as possible: close any non-essential apps, unplug internet cable and each time I have the urge to stop reading and go research a newly discovered subject, I remind myself that I am only allowed to do it after finishing what I'm currently reading.

Another helpful thing for me is to remind myself what I'm trying to accomplish with whatever I'm doing at that particular time and what my long-term goals are.

For instance, if I'm working on a project for a client, the goal is to get the work done as soon as possible and obviously get paid. I am not working on said project to primarily enrich my knowledge, but to make money.

I can use the time after the project is delivered to draw conclusions from the experience or do further research.

This may sound trivial, but it really does help to constantly remind yourself of what your goals are, it keeps you in check.

For extra effect, every time you have the urge to let your mind wander too much, try imagining the possible consequences of not completing your task (on time).

This can be particularly effective if you're doing client work. Imagine how embarrassing/unprofessional would have to explain to your client/employer that you won't be able to deliver on time because of something that you could've prevented.

Side-note: you have not provided enough info, but you may have obsessive-compulsive disorder, which is nothing to be ashamed of, but you can only 'solve' this with medication, so you would need to see a doctor.

Two ideas: (1) Find a partner for each side project, or at least 1 target user (real person you know) you commit to serving by a hard date. It's an uphill battle to maintain accountability without another human. (2) I draw a box on my whiteboard and write in it the essence of the project and only the 2-3 deliverables essential to achieving it. When I take on a new activity, I challenge myself as to whether it goes in the box or represents a branch from it. Forces me to visually acknowledge how I am spending my time (and thus define and stay to the essence of the project).

Discard the unnecessary. It isn't always easy to tell but there are often subtle signs around what is and isn't necessary. Unnecessary things usually feel sort of shadowy in your mind or require an additional effort that doesn't fit square into what you are doing. Think about what it's like to carry too many groceries into the house at once. Necessary things are solid and leap forward into your mind with evident emphasize. Be ruthless with your pruning of those unnecessary branches and your trip down the rabbit hole will be both smoother and more thrilling.

Great post. Thanks for taking the time to write this up.

I think you're probably not doing projects to help people, but instead projects that you think would be fun to do. If you start with a burning itch, or a person that needs help, you will probably find it much easier to stay on task. This is not to say that there's anything wrong with exploring interesting technologies, or thinking about "the right way" to solve certain problems you find interesting... It's just to say that when there's a burning need the interesting diversions tend to fall away.

Hope that helps :)

It's not necessarily that what you're doing is the interesting option; I'd say that it's the easier option instead.

Reading about interesting stuff is deeply satisfying, and so it's something you want to justify so you get to do more of it. Actually getting stuff done is hard.

It's a bit like why so many techies - myself included! - vastly prefer to code new app features rather than working on marketing or sales copy. One is has a definite end point, and we feel comfortable getting there.

The other one can seem like work!

If it's a side project, you have to ask yourself what kind of side project it is. If it's a business or something you want to turn into a business, just take any task that takes longer than an hour or two, put it down on a notecard or sticky or some digital equivalent thereof, and put them in a priority queue. If it's just for fun, don't worry about it--if you're not frittering away on things just to please yourself, why are you hacking anyway?

What happens when you implement a dreadful, obvious, hacky solution? Do you find yourself compelled to change it? Or do you not even get that far?

It sounds like at least part of your problem may be perfectionism. I did a bit of research on this a while ago, and it turns out there's a lot of literature on perfectionism and how to manage it. A quick look on Amazon under "Perfectionism" should bring up a few interesting books.

It's hard to generalize. I think the amount of perfection I demand in a solution is proportional to some combination of its importance and my interest in it. I have no trouble putting in place simple, hacky solutions with the intent of changing them later, and then actually changing them later.

My side projects, in particular, are where I allow myself to roam freely, and I often end up prioritizing unimportant (but interesting) work that is only tangential to the core idea. When I am only accountable to myself, then it becomes too easy to get unfocused.

Perhaps the original question should be rephrased as "how do I keep myself focused when working on side projects, since I'm only accountable to myself."

Depends on the purpose of your side projects. Is the main aim to explore things that interest you or is it to "ship" something specific?

If it is the former then perhaps you shouldn't worry and just let the rabbit hole lead you where it may.

If it is the latter then perhaps it would help to produce some sort of roadmap. Ideally share this and get other people involved (either people who will work with you on it or who want to use the finished product) and they will perhaps keep you somewhat accountable.

If your side project involves multiple difficult problems then perhaps you need to accept that it is not going to be a 1 person job to get it done too the quality standard that you aspire too.

> It's hard to generalize.

It's actually very easy to generalize.

The answer is endorphins. The harder the problem that you solve, the bigger the fix you get. Hacky solution is not the solution, you know it, so you don't get much. If you realize and accept this as the cause, the solution to your problem (which I think is very common between the programmers) is obvious.

God dammit, now he's just going to spend the next two weeks chasing down obscure journal citations about the psychology of perfectionism!

Just think pragmatically.

Whilst the desire to strive for perfection is in any craftsmen, ultimately what matters is getting the product to the customers and solving their needs.

Only after solving their needs should you start to indulge in the craft for what the customer cannot see, the internal quality.

I also sometimes describe this as "be lazy"... don't do anything that doesn't need to be done (for the benefit of the business and your customers).

I don't much have a solution to the rabbit hole syndrome, but at the very least I can point out that your rating system could be solved by a confidence interval made using a Bernoulli random variable, which is a useful and (fortunately for you) a relatively simple formula to derive and comprehend.

I decided on a logarithmic weighted bayesian ranking. I read about the typical bayesian ranking (where you have some sort of minimal required number of reviews), then modified the it to work with power-curve distributed sets of data (as #reviews tends to be). Works great and has a few useful levers that I can adjust as I see fit.

Have you by any chance taken a look at the Wilson score? It's a modification of the typical Bayesian ranking that functions much better with small samples.

Actually, come to think of it, I don't know if my previous answer of "get more experience" solves this procrastination problem.

I think it's actually "have kids" that solves this procrastination problem. With having kids you get so many constraints on your time that procrastination is no longer an option.

I have the impression you are subject of a typical mild ADD syndrome. Explore that rabbit hole and you may learn alot about yourself and see what people do to handle this. An obvious solution is to just do what you like and you'll be much more efficient than the average person in that.

I think it's about finding a happy medium. You certainly don't want to cut this passion for learning & consuming knowledge out completely but maybe you want go along the route of shipping your first solution and then improving incrementally based on further information.

Want to break the rabbit hole habit? Start your own business where you have to get things done. You'll either break the habit or go out of business.

BTW, going down the rabbit hole occasionally isn't a bad thing IMO. But if you go down every rabbit hole, it can slow you down.

Oh boy. In a sense I have the same problem. I've started so many projects but I never release them because I've always been afraid they're not perfect enough. It sucks I know. I guess that's why a lot of people propose the MVP, then continue from there.

as an engineer you have to realized there's not a best solution overall but a best solution for this particular problem. Each solution has pros and cons.

I think you spend time weighing each design because you're unclear on your final vision of your product. You haven't definitely answered what your values and needs are that you're trying to solve.

This may be a hard question to answer. This is why MVP are neat, you can get a product out to consumers quickly and use their response to develop values and a overall direction.

I find you if you think too far ahead, you don't leave any room for random disruptive opportunities that can occur in each step.

So know where you're going. Figure out and focus on the next step that gets you close to that.

At what time do you work on your side projects? I found out that if I wake up early and work on them before going to my day job I tend to be much more focused. That might be just because of the effort it requires to do so :)

Just remember that, most likely, not a single user will ever be execute this code.

This sounds like a good thing.

Over time, your rabbit holing will develop in a direction called a 'specialization.' Take note of which areas interest you and what areas have opportunity and rabbit hole that way.

Honestly? You don't care about your side projects. You care about the puzzles that make up your side projects. If you want this to stop, you need to find a side project you care about.

Short term hack: medication.

Long term hack: meditation.

How long with each, how much of each... that's going to take a lifetime to learn and figure out.

"It's not enough that we do our best; sometimes we have to do what is required." -- Winston Churchill

Possibly you are not that interested in the main topic? Possibly adult ADD?

I do it and don't try to stop, cause it is good to learn.

My mantra for this is "only solve problems".

Discipline.

I have been thinking about this a lot lately. It's been an immense problem for me and it's destroyed almost all my side projects.

The problem is that you're interested in the wrong aspects of your side project. You're interested in learning and solving the code problems that come with your project. You have to shift your perspective to the human side of your project. You have to learn to see the user's issues as the problem to be solved.

Let's use an "reviews" site as an example. Your users cant find reviews, that annoy them. What you're solving is not a programming problem; your problem is "make it easy for users to find reviews". That is your problem. Your problems is not what the best algorithm for ratings. None of the code stuff matters. These users do not care about code.

When you've shifted your perspective to the human side you'll start to see hacky solutions differently. A crappy algorithm for ratings isn't a poor solution, it's a perfect solution. You've solved the reviews problem. You have a perfect solution.

The problem is that most of aren't actually excited about the human side of our side projects. This is because we get excited about things that interest us, and what interests a programmer is usually programming. It's not surprising we choose projects with interesting code problems.

Start reflecting on the human/sociological side of things before you choose a project. You might be surprised at what you find. Projects that seemed interesting may suddenly become dull and the one's you thought were dull might suddenly become interesting. A human perspective doesn't exclude building stuff with a code focus, just make sure coders are your audience. You and your users just need to be excited about the same things.

There is this classic saying about "building something for yourself" resulting in the best products. As with most sayings, they left out some important information. They forgot to tell to you to make sure that what you're building for yourself is the same thing you're building for your users.

Make your goals align with your users. Otherwise, you'll find yourself trapped down the rabbit hole.

Some extra thoughts

Don't forget that all your tricks to solve programming problems work on human problems. For example, for a reviews site it's easy to phrase the problem as "make it easy for users to find the best reviews in the most efficient and enjoyable way possible while allowing them to simultaneously book and view and and and....". Break it down into it's simplest elements first, just like you would a programming problem. The simplest element of the problem is "make it easy for users to find reviews"; best reviews are a separate problem.

You're probably good at solving programming problems, breaking them down and being productive. You know all the 37signals posts, all the design patterns, and can quote re-work by heart. Use those principles you have learned and apply them to the human side. Everything your learned about productive programming applies to productive life.

I have like 20 ideas in queue I'd really want to prototype and they are of every kind... website, business, writing, drawing, mobile apps, artsy stuff...

Also sometimes feel I read way too much

One post that really catched my atention that's related was: http://www.ribbonfarm.com/2010/03/18/the-turpentine-effect/

I related to it in the way that I find it difficult to see myself as a super nerd technical robocoder, I need to create too, can't not think about creating and ~wholism~ in this because it's the reason I started to learn it in the first place, so I accepted that

So, well, I'll buy a board, outline stuff and start on them, maybe I'll do an Output Week in that I'll try cranking projects for a week and this is good because of learning and etc like other ppl said here, but learning how to focus is just as much important, so is actually getting-things-done... so it's like "there's no problem in running as long as you know where you're going"

I think the anxiety increases as you don't do stuff and the more you do the more you also learn what's worth doing, no? Before I tried setting out to do a complete project by myself I'd probably accept the invites of "Hey let's start this project, you code!" more, now that I know the work it takes I'm way more picky because I know it doesn't matter if I build a complete skeleton for this but then there's nobody up to even do a fucking design for it and I won't be willing too... so it rings "not worth it"

If it's something that's clearly not ever seeing sunlight, if it won't be at least instantly useful for me, if no one is buying/using it(in the future)... why do it? even a painting has to have a meaning and a purpose

Aren't the best actors the ones that are picky about work they'll take? Aren't the top industry guys who get teased by Facebook, Google and most promissing startups picky? I'm pretty sure they are... I'd guess it's because they know they are basically unable of half-assing stuff, so they need to choose well what they'll set out to do.

I'd say that's the positive version of people with this personality trait, the negative version of people with the same personality trait are the anxious, lost, starting-a-new-thing-everyday-and-never-finishing-anything guys... so IMO it's something for each person to work out

Reply on news.ycombinator.com