Hacker News Reader

Have you ever been in a situation where communicating and breaking things down takes longer than the actual work?

IMHO most disasters I’ve seen occur when ‘Product’ tries to strictly control the ‘Developers’. It just does not work for certain kinds of projects. It also diminishes the developers’ involvment and leads to a culture of mediocrity and CYAs.

Also, a person that does not have to do the work will always be waaaay more optimistic about how much it takes to do the work.

How do you fix it? Trust and ownership. Product and Dev need to have an interface - a boundry if you will - and the discussions and talk are about defining this interface and setting expectations. Product trusts Dev on implementation. Dev owns the delivery and makes it happen.

This is why agile tries to push the business stakeholders to define -problems- rather than -solutions-. Developers then try and come up with a solution, and pitch it to the business (effectively).

Almost every issue I've had with the business/developer dichotomy (and I use that instead of product because different orgs place product differently) has been due to the business trying to pitch solutions to dev that invariably end up not actually addressing the problem (even simple things like "that's the wrong color" or similar, are because they believe they're responsible for the solution, and they have a very clear idea of what the solution should be, without actually being able to articulate or recognize the problem. Does the color actually matter? If you have a problem that implies it does, yes, if you just came up with a solution on your own and in your head that solution was yellow, no).

Maybe, maybe not. I’m very suspicious of things that are pitched as a panacea for everything.

The real world is a messy place where even with a good team, a good plan and sensible expectations you can fail spectacularly.

If there is one thing that I really like is aggressive validation of the assumptions that are being made + keeping the product as small and lean as humanly possible (and the latter should apply across the board to everything you do).

Complexity and Entropy are preparing the coffin for your work even before you have the idea. You can give ‘em a run for their money, but in most cases you’re going to end up 6 feet under. As soon as you accept this and are genuinely open to failure you will be ready for the opposite.

Oh, I'm not saying it's a panacea for everything. Just that most issues I've seen along these lines were also accompanied with a perspective on the business side that their role was to give solutions, which were invariably insufficient, led to the dev team just doing the minimum (since they had no idea what the core problem was and thus were unable to determine where 'sufficient' really was), and that compounded.

That definitely happens and it’s... unfortunate

> Product trusts Dev on implementation

Currently having issues with this one right now, and it is the most frustrating thing I've experienced at a job.

The problem is that product and dev are different people. Product managers don’t understand their product as well as the devs do. devs should manage the product definition and work with feature requesters at a very high level, where the details are the choices of the devs (or senior devs acting as product managers.)

you're not alone. I would say that lack of trust is the norm not the exception when it comes to software dev teams :(

Ditto. A colleague called into question an off the top of my head estimate. A 2 day estimate and tried squeezing me to cut it down more... For work I wouldn't even be doing. Nope. Not going to do it. I tried to call into question his estimate of how long it would take for one of his sales guys to sell the lead he received yesterday. Once he threw out a number I asked why so long? Needless to say the conversation didn't end on a good note. Live and learn. We'll talk it out tomorrow. A quick way to lose trust and respect is implying incompetency and squeezing/trying to manipulate people. Really kinda pissed at him right now.

I have to respectfully disagree with #1.

Software is hard because we 'don't know the unknowns'.

It's not like building a building - if it were, we could just schedule it down to each detail. We would be still all be using 'waterfall'.

If we knew how to write the module, we'd just do it, test it, and voila.

But most of our time is spent 'figuring stuff out'. Oh, product has asked for a tiny change (support a specific kind of font), but our libraries don't support that, meaning major possible architectural changes.

The other major unspoken factor is that requirements change. In fact, this is the #1 thing, really. Requirements change because customer's needs change, because business changes because our understanding of the market changes, etc., and this is a reality of our world. So the nature of how we're going to adapt is difficult to predict.

This is why the #1 predictor of outcomes is the quality of developers on the team i.e. experience, intelligence, know-how of the domain.

If you're doing something where there are very few unknowns, well then it should be possible to schedule within reason.

It's not like building a building - if it were, we could just schedule it down to each detail. We would be still all be using 'waterfall'.

I hear that it doesn't work that well for buildings either, as you end up with people in the field detecting bugs in the plans and having to improvise fixes without input from the designers.

For actual buildings, they spend a fortune up front planning.

Soil and dirt samples at $100K - just to do a bid on a project, let alone the actual build.

Can't spend $500M on something only to find you have to tear it down.

Software is not magic, but it does require thoughtfulness and experience ... (hint to all those who think startups require mostly new/young blood - it's mostly wrong, but that's for another post)

> The other major unspoken factor is that requirements change

Walking on water and developing software from a specification are easy - if both are frozen - Edward Berard

Also when the dependencies are frozen. If these change, you're walking on mud. It sucks.

Comparably it'd be as if building developer built from prefab that changes the design every week.

> The way to do this well is to break down a feature until they’re at about half-day sizes tasks.

I have never seen that work in practice.

Breaking a task down into meaningful half day task is almost as hard is just implementing it, but usually this is attempted in a committee setting. This is not a recipe for success.

Agreed. Think of how ridiculous it would be to make a half-day task for figuring out X -- sometimes I've spent days, even weeks, trying to figure out how a module works.

In practice, being able to break things down meaningfully and accurately to such a minute level implies that a lot of time was spent in grooming, which means we're just back to waterfall again. (Or the tasks at hand really are just that simple and/or the people involved have a lot of expertise.)

I agree. Saying, "Break it down until it's in half-day size tasks" is fine. Some of the things that might interfere:

1. Are you sure you can do that in half a day?

2. You forgot about this one part of the task that you thought would take half a day.

3. There was a bug in the plugin that you were going to use to do this task in half a day, and you spent three days fixing it.

4. This task will actually take longer, but management won't buy that. Looks like it should be half a day on paper, so that's what we'll say.

5. I've never even done this one thing before, but it sounds like it should just take a couple of hours.

I understand the concept - trying to say something will take a week is even more error prone. But I think most of us are still pretty bad at estimating a budget for tasks even at such a small scale.

I've never even seen it work in theory.

When I read that in OPs comment I immediately had to think of "How to draw an owl" (the meme version, not one of the serious online lessons -- https://pbs.twimg.com/media/DcC1YEyVQAA9Lkw.jpg). "You simply break the task down into all its parts, then you just do it. Projects are easy."

Not to mention that that approach only works if you produce another version of something you already produced many times. There certainly are such development efforts, for example, I think anything data- and form based, like data entry + database + some reporting, what used to be paper cards a long time ago, and then punch cars, those type of apps can probably be made using that method.

So that's when they had to build the freight version of the Boeing 747. Then they thought "that was easy", and started the Dreamliner, something very different that they had not done before... (and it even still is an airplane, so not even that outrageously different as different pieces of software can be)

I think the "break it down" approach also faces the problem of the "coastline length measurement".

I'm also reminded of an anecdote from my early days with computers: I had to write an assembler routine (as part of an 8 bit assembler program) that seemed too hard, so I outsourced it to someone else. He didn't know what to do either, so I explained it to him. When I was done I realized I had just implemented that difficult function - I wrote it down in half an hour and that was that. Planning well, finding out what to do, is harder than the actual programming. Again and again, in all projects, I find that the >90% part is finding out what to do. There are stories from projects that lost all their code, years of work, and they thought they were done and finished (business kaputt) - but when they rewrote the software they fond it took them only weeks or instead of years. When you already know what you want and how the end result is supposed to look like (and feel and behave) then writing it in code is the easy part.

> Planning well, finding out what to do, is harder than the actual programming. Again and again, in all projects, I find that the >90% part is finding out what to do. There

So you've advocating an approach where you... don't plan? I'm confused.

Perhaps you're getting too hung up on the half day part? That measure is a useful forcing function that ensures the dev team has thought through the approach in enough detail to identify where the risks are and what smart trade-offs can be made.

http://dilbert.com/strip/2015-06-07

I'll go against all the other answers and say that I have seen that in practice. I'm even doing that in practice, every day of the week. A half-day task would even be a big task, most of our tickets are a couple hours maximum.

We practice a lean-laced Scrum with 1-week sprints. And yes, it works. And yes, we go fast. So yes, it can work (thought we have several years of refining our methodology and have a big culture of pragmatism and problem-solving).

Do you mind putting out a number of hours (per dev team) that get spent in Backlog curation, story generation, and other budget planning every week? I think perhaps an important missing ingredient for getting budgets right is actually spending the time to do so. The smaller the company, the harder it seems to justify.

Indeed, for a lot of us in smaller companies, getting our clients to actually consider taking an agile approach at all is a stretch.

For a team of 3 devs + 1 scrum master: - Backlog refinement/grooming: 1.5 to 2h, 1 dev + SM (+ PO) - Technical refinement: 1.5h, all 3 devs - Sprint review + retrospective + planning: 4h, all 3 dev + SM (+ PO)

Story generation is done by the PO, that's most of his job (though we split stories during backlog refinement if needed).

I don't know if it sounds like a lot (I feel like it could), but turns out it's actually efficient, every minute of each meeting is productive because we know exactly what we're going in for, how it will go and what you want to get out of it!

> The way to do this well is to break down a feature until they’re at about half-day sizes tasks.

You're identifying what the outcome of good planning looks like, not how it's done. What prevents this is that teams read "agile" as "make it up as you go along," when in reality if you're spending more than 10 minutes discussing a feature, make planning that feature a sprint task.

> In my experience most deadlines are movable and a strong PM/Eng team can convince management to push a deadline back.

It's not generally the team in that room discussing those deadlines, though, it's software development manager and product manager. The worst managers are software engineers who get promoted to management, and when they get into those meetings they look at their shoes and are railroaded by upper management.

You really do need a manager who has the intestinal fortitude to go to bat for you.

My god, this is spot on.

Re #1, this is kinda the system 2 vs 1 thing in the article. I was lucky enough to have a PM drill me into this. More specifically: The bigger a task is, the likelier it's going to go over time. The bigger a task is, also the less likely each bit is well specified and more likely that there are significant portions glossed over that you'll discover during implementation, which will delay things and/or result in them being done poorly. If you're having a hard time breaking the task down, it's probably because the task isn't well defined enough, so go back to stakeholders and define better. In this model, planning takes a lot of time and effort. Also in this model, research-y tasks are different, and instead get a fixed timebox after which you have to re-decide how much more time to spend on it if at all.

Re #2, part of the problem here I think is that PMs aren't trusted / empowered with that ability or final say. "Stakeholders" get to make that decision, but with ignorance of what it's currently like in the trenches.

I'll also wholeheartedly agree with this comment—my company has a long history of shipping on time and under budget. The caveat is that we closely follow #2: we aggressively strike requirements from the overall target if it means preserving the integrity of the project overall, with a commitment to return later.

I'll add this: a product team that ships earns 1) more credibility, and 2) that credibility is reciprocated towards the team in the form of greater flexibility and freedom to set their own targets.

This approach does not scale to maintenance. You cannot strike down anything when maintaining and/or porting. (And it is risky to even cut testing.)

How do you estimate how much time a task will take, especially when it's a novel idea, at least for your developers?

You don't. That's the thing. Yet decision makers commit to deadlines anyway, and that's the problem.

You break it down (effectively doing the work) as part of planning and after that you pretend that you have a working system. Make sure not to go into details and/or “take it offline” until the developer is tired/annoyed enough to just agree with you pipe-dream estimates /s

I see, so you simply hire superhumans who finish the project before UX mockups are even available. Simple stuff. I think I'm ready to start my own consulting business

You got it. Make sure every task can be done in half a day or less. That is key /s

Invest time in determining the right approach and planning it out.

Most of that will be doing spikes: http://agiledictionary.com/209/spike/

It's one the most underappreciated agile concepts. I wish people would shut up about standups once in a while and give more thought to doing spikes.

It wouldn't be easier if we didn't have to keep learning silly new words for everything. I used to do prototypes and proofs of concept, now I have to do spikes.

Spikes are not necessarily proofs of concept or prototypes. A spike is simply a task aimed at gathering information.

Spending 30 minutes researching four different libraries to see which one to would be a spike, for instance, but it doesn't necessarily involve building a proof of concept.

> How do you estimate how much time a task will take when it's a novel idea

In my experience, they just pretend it doesn't exist and remove it from the "plan". Where I see this the most is in figuring out how things work. For some reason, non-programmers have it in their head that once you've finished college, you ought to know (even though they don't) how everything that goes on a computer works instantaneously. We're going to use Apache Kafka! Ok, need to spend some time figuring out how that works. Um, ok, that should take like an hour, right? Because there's no time in the budget to learn anything. If you push back, they say, "ok, never use anything new that you have to learn. Only use the things you already know". That doesn't work either, because the things that we have to integrate with don't work with the old things. Or, "you don't need to know how that works, Bob knows how that works, just ask him. But don't let it take any time, because there's no time in the plan for that" And Bob is sick of all your questions that you'd know the answer to if you'd just spent a two or three days reading the online documentation and installing a test system and testing it out.

You hit on one of my favorite gems, "Bob has done something like that before, just ask him how to do it." The manager is envisioning that the team already possesses the knowledge necessary to get the task done, so there should be zero learning time involved. What this is missing though is that Bob isn't going to be solving the problem for the current task AND/OR Bob didn't even solve a similar problem, he just used the same tool or library that is now being used for the current task. If we had a Matrix like ability to instantanously transfer knowledge then this approach might work, but we aren't there yet.

I'd add #3: over-engineering. 90% of the time a simple quick solution will be as good as a super sophisticated and abstracted solution that will be delivered 1 years later, when the underlying problem it is trying to solve has changed (cf the usual tree swing cartoon [1]).

[1] https://www.tamingdata.com/wp-content/uploads/2010/07/tree-s...

Sometimes. And then someone adds layers and layers of crud on a "simple thing" because nobody actually engineered it.

And then some poor soul gets to rework all this...

Basic abstractions help a lot. Specifically, having things weakly coupled.

Moving milestones, incomplete requirements, too many meetings, wrong tech stack, trying to break everything into halfdays to fit an agile agenda.

What does work is waterfall. Whatever is in the repo at the end gets shipped.

With the exception of video games, I maintain that it's always better to be 10% over estimate on every milestone than to all of a sudden be 50% off of the final deadline.

To work it takes being comfortable saying "that's my/our fault", and my experience is that a lot of people who would rather die than admit fault. It's part of a larger raft of skills around transparency. So they cover up problems hoping for some miracle that never comes (or more cynically, they hope to be employed somewhere else before the shit hits the fan).

In the context of a fixed-time, fixed-price/budget project it will fail to meet these constraints unless the scope is very clearly delineated at the outset, and ruthless scope/change management is applied throughout the progress of the project. And if at some point someone realizes that there was an omission in the scope definition, the time and budget must change.

If the client or management want to change the scope, in any way, the discussion must immediately start with questions such as what other feature are we dropping, how much time are we adding to the delivery plan, what are the consequences on other features or the current design and code?

Even deciding to remove a feature can cause delays (e.g. having to re-structure a test scenario). Or changing a string in the user interface, which may require re-arranging the layout, re-translating the string, suddenly having to manage plurals, checking that a variation of the old string wasn't already used elsewhere, etc.

Well sure, if you doom the project before you start, with a fixed price and scope, it will fail.

The original primary point of the Agile Manifesto is to stop doing that. There's no excuse for it, unless you are required by law as a government contractor, and in thar case no one expects a successful result anyway.

I've seen and worked on many successful fixed-time, fixed-price projects, medium- and large-scale, not for government.

If you have deadlines, you already know management is incompetent.

Deadlines presume a known fixed scope. But I’ve yet to meet a CEO or sales team or whomever gets to decide what features go in who will fix the scope if a project.

They are always willing to add features, but can never take them out.

Wholeheartedly agree with #1. I'm fond of saying, "If it takes longer than four hours, you probably haven't broken it down enough. Find the monsters hiding inside your big tasks."

I’ve seen concerns and FUD over getting lost in too many tasks. How do you manage the overhead of 20 4-hour tasks against 2-3 high level tasks?

Well, they're just checkboxes inside the main tasks. You just think through how you're going to perform the task (which you'll anyway have to do when you get close to actually doing it), and note down the sequence of steps. Then you try to estimate each step. Sounds simple, but it works.

Why bother estimating? you have the checkboxes, now just get to work. If the pace of checking those boxes is slow then you can start cutting things out. Estimates are for managers, much better to just ship it already.

Because if you haven't built it yet, you can't ship it already.

But you are right that estimates are for managers. Managers should watch their reporter's progress and use that as evidence to build evidence. If the first task of 20 took 1 week, it's a 2year project. Update estimate as time and progress and work remaining increases.

How do you create this detailed plan? Are there any tools or frameworks?

There are literally hundreds of methods and tools; it doesn’t matter which one(s) you pick.

I’m quite serious.

The thing you need to realize is that you’ll get it wrong, no matter the tool... until you invest time in reviewing past estimates and plans.

People think because they’re engineers they can just magically make estimates... but its a skill. If you don’t have a feedback loop to measure your progress and adjust your actions, you’ll literally never improve at it.

...that’s why even ‘Senior’ developers and ‘technical leads’ are often terrible at this; they’ve simply never bothered to seriously make an effort to learn how to do it.

I'm a big fan of arc42