I understand product owners and businesses wants estimates but software engineering is not making a car. Forcing programmers to estimate their every task makes for a very bad work environment. They are not robots.
It requires more project management, but it's much easier to quarantine problems and explain them to stakeholders.
My rule of thumb is 4 hours. If you think it will take longer than 4 hours, it should be broken down. Ideally, most tasks will have estimates of 1 hour.
That way, when a 2-hour task requires 2 weeks to complete, you can point to the specific task and its challenges. It provides an easy explanation for why the release was late, why the requirements were too ambitious, etc.
Engineers do not need to be managed down to the hour. Good grief.
The micro-tasks I'm describing go into a project management system. They are either assigned to or picked by developers who complete and document the micro-tasks on their own. Because the tasks are so small, there is almost no management of developers required.
Finally, it's trivial to find out where project estimates went wrong, because any hidden challenges are well-quarantined in the micro-tasks. If the task isn't well documented, it's still pretty easy to go to the developer and find out why it took longer than estimated.
If this isn't micromanagement, what would you define as micromanagement?
I think some managers just don’t have enough to do so they make work for themselves (and everyone else). Like a border collie with no sheep to herd, they will start herding ducks, children, etc.
If development was so easy that a manager who doesn't understand programming can break developer tasks down into small chunks - we'd live in a completely different universe!
In reality, management breaking anything down into smaller tasks, instead of people who will actually do the work, breaking things down, is going to lead to ZERO architecture and a shit codebase, 100% of the time.
That's if you're lucky and the project is a clean slate.
Existing codebases are full of code smells that'd take days to comprehend, days to chase down and comprehend what the actual requirements were/are because those are never properly documented, and then re-write that 'micro-task' so that the next developer doesn't have to spend their days in hell.
Oh, this'd need to now be tested, there'd be new bugs, and those would need to be fixed also.
So instead, in the real world - your micro-task gets handed down, the developer doesn't bother understanding what the existing codebase is doing, they just patch it until it works, contributing to the unmaintainable piece of garbage that all developers with any self respect left in the world hate.
Since only the brave/crazy ones will say 'this micro task will take 2 weeks instead of 2 days because the existing code is garbage', those brave ones will get replaced by those who will say 'yes, 2 days'. Your project will need more and more time per task, and more and more developers doing maintenance, but the managers will continue reporting successful micro-tasks completed on time!
I've just described corporate programming and how you're directly contributing to the hell that it is.
More precisely, a no-development-management-knowledge manager’s worldview.
While obviously development experience can be a source of development management knowledge, there's no reason it should be the only source, and development managers ought to be screened for job skills as much as developers are.
That's why McDonalds works, and software development is a perpetual disaster - McDonalds understand that to manage even something as simple as flipping burgers, you need to have done it yourself first!
It's the arrogance of business school management that's responsible for a great deal of turmoil across many areas of everyday life. Just imagine going to school for something that's not difficult, to 'learn' how to boss people around. For good pay!
Isn't that the dream of every non-creative, lazy, half-wit you never want managing anybody ever? Yes... yes it is...
This is a recent phenomena on a mass scale and it isn't going to last - getting high rank without having earned it has always been a complete disaster, just look at any point in history.
This is just another way of saying management is incompetent. There's no project management strategy that is going to overcome incompetent project managers.
For the engineers, and yours, peace of mind and progress -- please spend some time with stellar execution managers that understand software development is a people problem, not a project management problem.
And I agree with you: talented people, be they managers or developers or janitors, will be able to overcome any process or planning shortfalls.
But there's no difference between what I'm describing - a project manager breaking down features into bite-size bits of work - and a developer taking on a single feature and writing down TODOs in code, on post-its, on a white-board, or in the comments of the project management software.
The difference is one of documentation and communication to stakeholders - which is the PRIMARY problem with software estimation. There are ALWAYS unknowns in software. A team leader needs to be able to go to his boss when something is falling behind and say, "We didn't anticipate this. This is the reason it's taking longer".
This is FAR superior to going to the developer and asking them why feature X is taking so long and having them compile a report from Notepad/post-its/white board scribbles. With my process, you can just pull up the project management software and find the tasks that went way over, and the developer stays almost completely insulated from management.
I'm sorry, but you haven't really worked with or had training from real-world engineering managers.
This is the worst kind of condescending ignorance, it betrays an inability for civility that I wouldn't tolerate in a colleague.
It is not your fault if your leadership demands detailed answers as to why X is off by Y days. All you're doing is cascading that mistrust/poor management skills to your engineers. I understand what you're doing is the most reliable way to do it, but there is a better way. This is why I mentioned its a people problem.
The better way is to not break down things into hourly tasks but weekly sprints as most teams do. Let your engineers break that down into how much ever granularity they're comfortable with. If something goes off track, build communication trust that helps them keep everyone in the loop and work towards a common goal.
As a product person, I've had the fortune of working in two of the big 4 with some of the best engineers there is - so maybe that contributes to my worldview and it might be terribly biased.
Thanks for explaining :)
However, as padobson notes, there's an overhead associated with breaking complex things down into smaller units. There's a point at which it becomes counterproductive.
I feel like I have, over the years, sometimes crossed that line.
(And, other times, not gotten close enough to that line and been surprised by weeks/months of schedule disintegration.)
The only thing that makes small estimates seem better is the worst case tends to be less time, but overall they don't help.
Giving single numbers for an estimate with uncertainty is bad for all parties.
Additionally, with experience one can get better at giving estimates with less uncertainty for well known tasks and recognize the uncertainty for other tasks.
AKA Y is much higher when you deal with low hour estimates.
This is most noticeable when you have lot's of tasks. If you think 200 tiny tasks may take 1 month or just over 1 year that's an almost useless estimate.
I wrote down in detail all of the tasks involved just to help me estimate like I always did. By this time we were transitioning company wide to scrum.
Every task I put in was scrutinized by the contract “scrum master” as it wasn’t part of the MVP even though I knew it wouldn’t affect the schedule since I had prior work I could base it on.
The rest of the dev team had the same issue. I got tired of fighting and the scrum master was getting in our way. I told them to do what they thought was right and run it by me first if it was anything major. None of the things we were trying to do were customer facing, they were all backend architectural things.
I tried to fight the good fight but then had the epiphany - I didn’t have to. I’m the dev lead, I do the code reviews and no one higher than me will be looking at the code. I told them not to be detailed in the how within our project tracking system.
Another senior dev told me the same advice you realized. Now we just create more generic stories. He’s stared to go story by story and wants to know how it meets a business objective.
Our stuff is extremely customer facing and time critical. I welcomed Scrum over Agile but a bad Scrum Master can kill the benefits.
If you don't know a given approach will work, sometimes that's best to timebox some experimentation time (say an afternoon), but then you can also say to your PM/EM 'ok, we need a little time to work out how long it's going to take', and decisions can be made accordingly as to priorities.
Very little of being a software engineer is about writing code. That's just typing. The skill is being able to come up with a coherant plan to solve a problem. That's why architects don't tend to write a lot of code.
I use the term discovery phase to describe this. You give the discovery phase a certain amount of time to complete, it should be from a few days to 2 weeks. The deliverable at the end of discovery phase is a document. The document explains what experiments were run and what we should do next.
But I know plenty of people who need constant monitoring and handholding.
This also goes well with the rule mentioned in the article: always have at least 2 people working on a project.
It doesn't make things faster but instead takes a lot of mental energy that could be better spent on solving the problem.
Frankly if I'm going to spend a couple days on one aspect, I'd rather just implement that aspect.
I prefer to identify the risky parts and do them first, and refine estimates along the way. TDD helps because it keeps components isolated, so you don't have to build "in order".
Waterfall development with adequate time for planning/discovery up front does for sure give better estimates -- but those estimates are very expensive to get and are usually not all that reliable anyway, especially for multi-month projects.
In other words, "anything useful".
My impression has always been that the true cost of a dev task is only revealed when solving it. That's why they're hard to estimate. If they were easy to estimate, they would be trivial to implement and therefore could be largely automated. And for the lack of work, we'd aim for the next larger goal (which is again hard to achieve and estimate).
This seems to be a concept so hard to grasp for non technical managers.
By definition tasks aren’t that similar to other tasks, otherwise we would be a super crappy software shop that can’t provide semi-generic solutions.
Dev team’s mission is to solve unique problems to the stack. Discovery is a major time consuming task by definition, and can’t be properly estimated, as it’s an information problem.
The problem with this is that the time required to complete all the pieces is almost never equal to the sum of their estimated durations.
Do you mean bank holidays? Cause regardless of how you feel about your company, you should always take those. Rest and disconnect from work are important, especially in environments where you really like the work or the job, and the risk of overheating is much greater.
That being said, your point about bank holidays is a good one. Never give your company free labor -- they'll never give you free capital.
Realistically, tasks should be grouped into roughly day-sized estimates for any long project with an equally large QA pool possibly run in parallel.
Granularity only gets you so far before the returns diminish to the point of worthlessness. Imho anyway.