Breaking down tasks
jacobian.org
jacobian.org
One, it turns out I never ever end up doing the actual steps as I planned to. After a few I have new insights, realize I forgot something, see an easier way, I don't know -- the plan is never followed.
The other is that I really hate working like this because it seems all the creative effort of thinking about how to make something is put in the first part. And then, the rest is still most of the work, but it's the really boring part. It's much more fun (and therefore faster, and resulting in better work) to spread the creativity and the boredom out more equally.
The two are probably related, and it's not inconceivable that I have ADHD.
The plan not being perfect and prescient is part of the plan.
Next time you plan for same or similar activity, your future plan will be better.
The Project Managers even have a mantra: "fail to plan is plan to fail."
But for me it's very easy to overdo it. Give me interesting chunks of about a week or two, with a tight deadline, and I'll do my best work.
Remember, Linus wrote Git in two weeks. And there was no project manager in sight.
Of course if you are exploring or building your own projects and/or don't have any other accountability, you wouldn't start planning that.. unless you are really into planning in and of itself.
But as soon as you have a boss and the boss asks "how long? who does what? where do we start?" - you have to have some framework.
I also have a very high aversion to "conforming to the system", esp. when these systems seem arbitrary or are overly rigid .. but systems in general need to be simple and flexible. Productivity systems are there to help people.
I don't think the author intended for this to be like a detailed recipe.. but more like an example of how he approaches these things, which can hopefully help others have their own ideas.
Why?
Does it work? Is it productive? Is it better than not doing that?
You still have a boss asking these questions. Any answer you provide constitutes a framework.
>> On average: yes, yes, and yes.
For what it's worth, I think it's more like: On average: no, no, and yes.
It doesn't really "work" in the sense that the plan turns out to be an accurate and useful reflection of the work; that is not the case on average in my experience (sometimes it is, but not usually). And for the same reason, it isn't usually a productive exercise; it usually costs more time up front than it earns in productivity.
But despite those two things I think it still turns out to be better on average, in a team / "you have a boss" environment. Because productivity is not the only thing that matters, legibility is also important to organizations, and worth some productivity cost. Though this will chafe me until I'm in my grave, I still think it's true.
Having people assigned to tasks, a starting point, some initial steps to follow is something that increase delivery speed in the beginning a lot.
Edit: to some extent, legibility is productivity, as it means others are aware of what is going on and can interact/refer/etc.
It's a balance of time spent planning against the value of the plan. Immediate-term "planning" is definitely valuable, IMO. Thinking through how some immediate work can be split up so that multiple people can make progress simultaneously, without stepping on each other's toes, that is definitely valuable. Having some discussion about what the next steps might be after those immediate steps are complete, that's also pretty much always valuable. Spending a small amount of time thinking about the next next steps can be valuable. But past that, in my experience, people often put more time into planning detailed steps than ends up being valuable. It's not that it has zero value, it's that it has a pretty high cost that, in my opinion, usually outweighs that value.
Also, just to note that I think spending time thinking about a "north star" vision of what a project is trying to achieve is also pretty much always strongly positive ROI.
It's time spent coming up with a detailed plan a few steps out from the immediate term that I find dubious. I don't mind other people spending that time if they want to, but I don't like spending my time in those planning meetings; I'd rather be getting to work on the immediate term tasks.
But actually, I kind of feel like people who have had training to get good at planning, are not the most likely people to be good at avoiding "overplanning". I think it's the same kind of incentives as how dentists have a natural bias toward recommending dental work, or chiropractors toward recommending adjustments, or programmers toward recommending writing software, or anyone toward doing a bit more of whatever it is they are trained at and good at than a totally unbiased third party might really need or want.
Maybe there's also a bit of a "midwit meme" to this (in all of these cases): inexperienced: don't do it!, some experience (midwit): do it!, extremely experienced: don't do it ... except in these cases where you really should, and include just the right amount, and be very thoughtful about the forum and who to include in the discussion, and calibrate the right level of granularity for this specific project, and ...
I would say that "a good programmer doesn't write more software than is necessary", but despite my best intentions, I'm certain that I write more software than is necessary, because writing software is my hammer and lots of things look like nails to me.
I think it requires more than just being "good" at something to overcome this tendency. I'm sure the very best operational planners, like the very best software engineers, consistently strike the perfect balance, but I suspect most "good" planners, like most "good" (even "great") software engineers I know, still struggle with this tendency.
I do the GTD thing of only thinking about the _next_ action, not trying to think of every action.
I.e. not somehow trick yourself into not thinking about it, but just think who knows if/when I'll actually get around to that, for now just do this. Don't worry about optimal order.
If the skirting doesn't look as great as it could because I fixed and painted the doorframe first, only painting the wall and rest of skirting later, meh who cares, if it turns out to be that noticeable maybe I'll get around to giving it all another coat; in the meantime at least I got it done, not stuck in 'analysis paralysis'.
You can plan all you want and it is essential... however you'll quickly find out that most of your plans are in your newborns' hands.
It's a framework to prioritize important tasks instead of falling into the agency trap, akin to prioritizing meaningful strategic tasks such as product development and tech debt instead of fighting fires.
Without planning, a depth first traversal is a high risk endeavor in the likelihood that the that path is wrong but backtracking and creating the graph is comparatively expensive and susceptible to sunk cost fallacy. Depth-first traversal is writing the book a chapter at a time without a table of contents in mind.
The plan to me is just a way of reminding my future self what I thought of as ideal output in the past. Instead of infinitely changing my plans and chasing fireflies
Realizing this also helps me avoid yak shaving over list making tools, since I could care less about even saving the lists! I keep a folder of text files in any project I'm working to hold my lists, but tbh it's unnecessary to even save them.
> the plan is never followed.
Sometimes it's even worst. As we add more tasks during execution, at the end, these tasks get twisted with the ones you created while planning. Then we end up with a messy list of undone tasks that aren't really required anymore (because they were created without having the full context, that only happens during the execution).
(A few times there is prototyping involved but I won't share that with non-devs, since they don't seem to get it.)
I think maintenance tasks may require more effort / more expansive qualification when it comes to the meaning(s) of "something is different".
I think the topic of "breaking down tasks" could very well be its own book genre or even podcast genre. My experience with self-help and organization titles has been that the activity of breaking down tasks is a sort of known primitive human operation that many authors assume their readers to have. However, my own personal experience and the accounts of others I have heard while participating in group sessions is that breaking down tasks can be very difficult and provoke emotions of avoidance and despair.
The most broadly relevant advice that I have come across when it comes to breaking down tasks is to keep breaking things down until I am 90% certain that I can successfully complete the task. The certainty here would vary on how much self-confidence an individual has and their risk tolerance, so some people may break down tasks to 70% certainty of success.
In this case engineers give an estimate of what they think management wants to hear and not the actual time it takes. It happens if they are frequently asked to reduce their estimated time to within management expectations.
To deal with this, you need to bargain with them in the opposite direction to make sure the estimates are realistic.
Kinds like the architect sitting between the client and the building contractor. The client wants the moon, on a low budget, and wants it completed yesterday.
A strong manager understands their role, and the constraints (time, money) of the system and is able to find compromise where necessary. They're there to guide the owner, and at the same time monitor the workers.
Of course the vast majority of middle management are not strong. They see their role as passing on orders from on high. The boss is the customer, and the customer is always right. So their only part is to demand more from the workers.
Strong tech workers understand what the manager needs. Good time estimates. Limited budget. The manager needs to be passing accurate data upstream. Bad workers ignore reality, tell a manager what he wants to hear, and just do their thing. Not realising that they (the programmers) will be the ones thrown under the bus ehen it goes tits up. (Another sign of bad managers is to blame the underlings.)
Good managers, working with good techies, is a match made in heaven. Together they deliver accurate estimates, with plenty of contingency time. Together they decide what features are necessary, and what can be canned. If you are in this dynamic, count yourself lucky.
If you are a good manager working with crap staff, well, that's unlucky. But at least you can turn them over. If you are a good tech working with a crap manager (which I suspect is most of us) then, well, I guess the only options are to stay, or quit.
Bad managers can be made better with explanations though. The more they understand the constraints, the better equipped they are to argue the point with the customer.
And also don't expect us to work 8 hours a day doing tasks.
Cook lunch, estimate 30 minutes. Task starts, oh wait, cupboard is bare, need to go shopping.
Over a long career I've found that you do the best estimate you can with the information provided. Then multiply by 4. Sounds extreme, but you're still likely to be short a bit.
Equally, for estimates, I like to first estimate the number of data tables. That's a good factor to bring in to the rest. A system with 10 tables, 100 tables, 200 tables and so on "magnifies" the task list.
"Build reports" has a proportionate number. 10 tables might mean 5 reports. 100 tables might mean 50.
Building reports might be easy, but the number of them means time is needed.
Me myself rely on Hofstadter's Law: It always takes longer than you expect, even when you take into account Hofstadter's Law.
Then they take about a quarter day each, provided the same person does all four simultaneously. Or about a day each if you give them to different people.
This has been amazingly accurate for time estimates - in aggregate... Individually, they are wildly inaccurate.
So... they are probability estimates. Over many events, they even out.
Not having a well defined final output and time is not very relevant for software, mostly because there is no per unit cost. Most managers do not realize this is they are not from a software engineering background. A versatile product can be sold to many different customers without additional development costs.
But since most companies are managed as factories, all the processors will ultimately create a very limited product targeted at a specific customer. What the big tech companies have avoided is this .
That would mean redo models that don't fit the whole piece, endless tweak some models that are really difficult to get right, sometimes do swiping changing across all the parts because as a team you realize something is off etc.
The balance will be between delivering something within a time frame, and delivering the quality that was requested/expected.
Do you have more examples of companies that are not feature factories? It seems the entire industry has adopted this pattern.
Changing the topic from software development to R&D: the granular task oriented view is generally not good for greenfield research endeavours. It's a form of premature optimization. You're calcifying the search space and reducing flexibility to pivot immediately based on expert intuition and hunches. That being said, a coarse task orientation is necessary to keep people vaguely pointed in the right direction. The best resolution (time horizon) for task orientation depends on what you're doing.
Too bad that's pretty much the only such example that has been touted since... The 1960s, is it? And that it was, IIRC, recently either shut down or announced to be shut down sometime soon.
Thus, using this as an example seems, if anything, to illustrate not that "there are too!" but "there are dwindingly few".
When it comes to less intrinsically motivated team members, I don't have useful advice. Maybe they can contribute to research output, but that would require more traditional management styles which I can't comment on.
When it comes to satisfying constraints imposed by other parts of the org, I can't usefully comment on this either.
I guess the essential problem I'm trying to solve is how to respond when the MD or a product owner comes to me and says "The client wants our core algorithms to be adapted in XYZ novel ways. When can you deliver this?".
Actually, none of them are. They're just managed as if they were.
In a factory, you can literally have a machine that stamps out 1000 widget blanks an hour and if the operator does an hour of overtime, that's an extra 1000 widgets produced that day.
You've got your workers on 8-hour shifts and sales signs a contract to deliver 8,500 widgets per day? Then you know right away you're going to need 10-hour shifts, or regular overtime, or a second 8-hour shift, or a second machine, and you can figure out how much each of those would cost, and whether you've got enough car parking for the extra workers.
Whereas in software, your feature factory produces ???? per hour, sales promises customers ???? and an hour of overtime achieves ???.
But honestly, I think what holds most of us back is a better ability to just don't do that. Take that thing you want to make, and instead of planning out all the pieces, just make the tiniest possible thing that could possibly be of value. Using this example, just start with "Today" and the 4 buttons. Make a thing that shows your 4 buttons of exercise you will click on each day. You can probably make that today. No streaks or freezes or calendar view. Those are all great ideas, and you'll get to them. But you need momentum. And you'll probably decide you don't even like that calendar view in a few days. I think more projects need a kick in the pants of just do something small. Get it done today. See what the next task is from that point because it's probably not what you thought it ought to be during the planning phase.
This also isn't exactly easy either. It requires some thought and discipline and find the smallest thing and commit to shipping it without distraction. But it's also something good to practice at.
> when faced with uncertainty, one must simply focus on doing "The Next Right Thing." [0]
This is not to trivialize it! Figuring out the next right thing to do is hard, but also the most valuable part of planning, in my view.
0: https://en.wikipedia.org/wiki/The_Next_Right_Thing#Synopsis
But thinking of it as "The Anna Principle", named after a character from a children's movie, remains way more fun. It was already a tongue-in-cheek-ism to ascribe what I really do believe is a strong foundational idea (not just in project planning, but in life) to a character in a children's movie.
By the way, I tracked this back to at least one earlier use of a similar construction (which maybe is implied to be the basis of its use in AA) by Carl Jung:
> But if you do with conviction the next and most necessary thing, you are always doing something meaningful and intended by fate.
Excerpted from: https://www.themarginalian.org/2021/12/07/carl-jung-next-rig...
But I suspect this is an even older, indeed timeless, idea. But ascribing it to a modern well-known movie character makes it more fun and memorable :)
I'm actually not saying every project needs this of course, but I think maybe most might? Obviously there's huge things I've been apart of, and we needed to be thorough planning step by step months out. But that experience should probably be the exception and not the rule. Minimums the rule.
Similarily during actual development modularization can be a way to parallelize development. So one component per person or per team.
For my own stuff I also like to interleave the human-facing parts of my products with the technological guts, especially if I am still figuring things out — developing the technology without knowing how it is being used will force the way you can use it into a certain direction — designing the interface without knowing the technology has it's own traps as well.
And at some point, you just gotta do it.
"spend 3 hours on investigating X, 3 on Y, 3 on Z"; and then have a planning session about what to do next.
You have 4 outcomes - the first one solves it, the second, the third or none.
If it is solvable with mild effort, you have a 50% chance of solving it by attempt 2/6 hours.
Although I don't even believe the real path to mastery is the task breakdown. Anyone can come up with a bad breakdown. The real point of mastery, which he didn't talk about here, is identifying when the evidence suggests a task breakdown is incorrect enough to cause problems and communicating that / doing a re-estimate [0]. And being comfortable that it will happen so not getting stressed up front putting out a schedule that is likely to change. Managers generally want an accurate roadmap up front. This is them asking for the impossible. IMO Good management is about flexibility and understanding that the expected nature of the work changes as time passes and developers learn.
He has a little example at the end. Ponder that if someone had come to him with an accurate estimate ("you can do this in a few evenings as a plane trip") he'd have rejected that as too risky. That illustrates that estimating isn't even about accuracy, there are many unspoken factors here around risk management, expectation management, task familiarity, etc, etc.
[0] https://jacobian.org/2021/jun/8/incorrect-estimates/ - he has a post on the topic
These are good, new software folks who genuinely don't know how to stand up basic functionality because the space is new to them. In my experience they want tasks they can learn from and excel at, leading to growth.
I think the thing that you absolutely must keep in mind is process overhead is a continuum you tune to your team. No one in the NBA is going into a huddle and getting instruction on how to throw a ball mid game, but the kids in third grade are. Both styles of planning and coaching are appropriate to the team at hand.
I think that’s the main truth about software engineering. Maybe an open source version of that Streak app exists and OP is re-inventing the wheel.
I think the brain doesn’t like task list. It may be pleasant to do, but most of entrepreneurship or coding is exploring.
And a task list blocks you from exploring.
Now, ask them to paint a mural instead. The estimates will change, they'll probably give you a broader range instead of being able to estimate almost down to the minute (and being off by no more than an hour or two) like someone just painting walls (with a known environment, surface type, and paint material).
Does the mural need to be designed? Are the desired qualities of the mural well-specified or will there be a series of back and forths? Maybe a set of prototypes (sketches) so that you can refine your requirements. At that point, the estimate for the total task (design and paint a mural, or design and develop a software application) becomes far less clear.
There are certainly programming tasks which are closer to the paint-a-wall task which are much easier to estimate reliably, but they're far from the only thing people in this field work on.
Customer: "Hi we need this small change." Dev: "No problem! Here's a quote for 4 hours"
Ignoring that 4 hours is probably less than the effort to consult, estimate, generate and process contracts, bill, do taxes... Then the customer comes back and asks for a revision which will at least require a Change-Order or a separate PO.
To some extent the overhead should be covered under labor burden but the reality is that in a sole proprietorship YOU are also the person doing the labor that is within the definition of "labor burden". It adds up extremely fast.
A million dollars would be a windfall if it landed in my checking account. On a development project with hardware and more than one dev it might as well be dust in the wind.
The seemingly infinite amount of variations in software tasks and the ambiguity of the requirements? If my job involved only putting up or modifying API endpoints that involved querying a single SQL database, I'd get quite good at that and be able to tell you with very good accuracy how long an endpoint would take to put up.
Just as if I painted many walls before, I could reasonably tell you how long painting a wall would take based on the surface area, height, and amount of non-wall obstructions. Also, contractors tend to be their own salesmen, so just because they speak confidently, doesn't mean they are. It's not like contractors are universally famous for being on-time and underbudget.
How long would it take to build a system to automate API endpoints that query a single SQL database? It depends.
The "manufacturing the artifact" step in software is done automatically by the computer, when given the "blueprint" (code).
I guess to try to go back to your painting walls analogy. In my view, creating software is more like if someone asked you to create a wall-painting machine that will work for any room that fits a certain specification. They could contract you to paint just one room in this way, but that would certainly be harder than just painting the room! But more likely, they have a million rooms they want you to paint. Either way, the hard part is creating the room-painting machine. And it would indeed be quite difficult to give an accurate estimate of how long it will take to do that. But once that exists, you can easily estimate how long it will take to paint any individual room.
And real engineers, indeed, have this same issue with the design phase of projects, for the same reasons. Just like us, they try to break apart and estimate how long it will take to do that part, but just like us, my impression is that it is known to be fraught to accurately estimate that part of a project.
But also just like us, this is a continuum based on how novel the thing they are engineering is to them. On the one end of the spectrum, there is the equivalent of off-the-shelf software, like using a blueprint for a standard single-family home that's been built a million times already. And on the other end of the spectrum are things like designing the motor for the first model of a new electric vehicle manufacturer, where it's all brand new. But then there's a whole spectrum between those two, where it is fairly easy to break down and estimate the design process for things you've done a bunch of times, and nearly impossible for things you've never done.
This is a very long-winded way of saying that the difference comes down to how novel the project you're working on is! This is why freelance / contractor shops do best when they find a particular niche of a kind of thing to build, and then find clients who want them to build essentially the same thing over and over again. It really is possible to get very good at estimating this kind of nearly-cookie-cutter work. (I did this a bit for awhile back in the day, with multi-page Rails CRUD apps, and it was indeed easy to break things up and estimate.) But this is also why it can be frustrating to work with freelance / contract shops, because it behooves them to figure out how to fit your project into their cookie cutter, and that can end up being a worse outcome than building something bespoke, iteratively, without a detailed plan and estimate.
If you try to make no unnecessary work, you'll end up having to do everything at once.
Example: you want to refactor a program with two modules A and B where B depends on A. The least wasteful way of doing it is to refactor both modules together. But it's also the riskiest and hardest to estimate.
The broken down way is to refactor A and adapt B so that it works with the refactored A. After that you would refactor A, which would then risk making the adaptation effort very short lived.
If you want to do zero throwaway effort, you often can't break down the task. I have 20 years of experience and I still often find myself reluctant to do throwaway/temporary job in order to divide work. Instead I find myself doing multi-week efforts with zero yaks left unshaved.
I mean, thinking through a problem you’re trying to solve before starting makes sense, and having rough milestones is important - but a majority of the time there’s so many unknown unknowns that fully breaking it down is totally useless, if not impossible.
If I spent the same amount of time it took me to break a project down just working on figuring out a solution or building it, it’d get done way sooner. At least, that’s how it feels.
Lot's of people feel like this. I usually come at it a different way - we're trying to estimate effort to see if we want to attempt building it in the first place. Assisting with the cost-benefit question if you will. That's the first reason.
It can also be extremely beneficial to the team, particularly when working with less experienced folks. You can farm out a lot of the work and parallelise the task.
I suspect Jacob didn't do much of this breakdown when implementing his streak app, which is why he recurses in an illustrative way.
It's absolutely valuable for managers to be able to estimate velocity as well, though the problem is that they really do stop understanding "we are not sure" once you spoil them.
I am a consultant too, so it's critical for me to say "we are delivering this, in this much time", even though we are "agile".
These are things I always share with them:
On code reviews: https://mtlynch.io/code-review-love/
On simplicity: https://grugbrain.dev/
On SOLID (even though we use Python now): https://www.baeldung.com/solid-principles
I would love to have a "standard" article for planning.
Similarly, if you don't plan enough, and get far enough into a small project, you don't change your approach because of the sunk cost fallacy.
I never warmed to that.
However, this is not that, and I think it's an important skill. I started my professional life as an RF technician, and we learned how to do this, almost immediately. We used things like signal generators and oscopes.
1. The "business" level. There are no "tasks". There are only needs and desires. Things like "a user can log in and press a button". These should not be broken down further than the smallest unit of deliverable value. In this example, there's no point estimating "user can log in" because it delivers no value on its own. A good rule of thumb is these should be described using descriptive language, not imperative, so not e.g. "implement log in procedure".
2. The "team" level. It's OK here to break down those things into "user can log in" and "logged in user can press button". That's because you know they can be implemented independently. But there's still no point delivering them independently. Don't tell the upper level about this breakdown. They are always really eager to know, but they don't need to know. Use this to calculate your estimate for level 1, but said estimate should be a single aggregate. Implementing these in parallel with multiple devs adds no extra overhead because they are completely independent.
3. The "developer" level. This is where you finally have "tasks" and imperative language. Things like "add button to form", "implement 2FA" etc. These tasks are naturally heavily dependent on each other and it's highly likely they'll be done in a completely different order to whatever you write them down in.
If you decide to distribute these tasks among developers then you increase the coordination overhead between devs. Think of it like a multi-threaded application. It's always more efficient for a single developer to do everything if possible (less overhead), but it might nevertheless be necessary to split it up due to time constraints, differing skillsets etc. It's just like we'd like to have one single 80GHz general purpose CPU, but in reality we have to make do with 16 5GHz cores split between performance and efficiency cores etc.
Given that the tasks are highly likely to change and evolve, there is not much point putting in loads of effort to break things down. Do it only when it's necessary, or when it helps you. It's necessary when you need to split up the work between developers. It's helpful when you think of more tasks during your work and you don't want to break your stack. Every developer should have a way to quickly take notes in a way that doesn't break flow, things like "ensure API accepts float input". This is stuff you'd never think of before you get your teeth into it.
Management are always salivating for details, but we know what always happens: we drop a technical term one time and management goes around repeating that like they know what they're talking about. They use these like weapons to keep other people off their back: basically say something that nobody understands and they'll leave you to it. Some managers will push for more fodder, but it's not your job to provide this. It's your job to deliver.