What Do We Know About Time Pressure in Software Development?
computer.org
computer.org
Your life is your work, but the ability to own 5 houses is a nice balm.
As an engineer I've experienced dumb deadlines, but I've equally seen wasteful decisions by engineers that are at best resume padding and often legitimately harmful to product and code quality ("let's make this a microservice" lol).
My favorite is when leadership wants something dumb or contrary to reality and engineering teams are happy to build it because they can use fancy technology. Then when the product is delivered late, a little buggy, and users don't actually use it anyway — everyone can shrug, point to someone else as at fault, then rinse and repeat the game. Incentivizing honesty is hard.
I'm so conditioned to providing buffered estimates, especially for highly-dependent systems, I would really struggle to not accidentally buffer. I'm generally invisibly factoring in organizational dependencies (e.g. Bob will need to stand up service X, but Aditya's approval of that will be subject ridiculous process Q that always takes two weeks). It sounds like this would invert the dependency tree if broken down well enough.
Cynically, I struggle to imagine working in such a high-trust environment that wouldn't immediately be destroyed.
> Every deadline can be met if employees are "eager to
> please" and will work unlimited overtime hours for free.
However, I agree with your overall point. Getting things done on time without abusing people (I consider using people's personal time to be abusive), with everything in harmony is hard.
It's easy to push your employees to log hours and have everything be a trainwreck when people are cranky, overworked, and cut corners to make deadlines.
I don’t think there’s a single right answer for that percentage, but the key is that it should be determined by the person or group defining the global buffer and should be done with consideration for the distribution of possible durations the project tasks might take.
In Goldratt's system, the risk comes from the longest (in terms of duration) chain of dependent tasks. Delays early in the chain can only be canceled out by speed ups in later dependent tasks, that's the primary motivation for that global buffer.
The technique was adapted from factory production line optimization. The case can be made that it can work well for projects doing repeatable and thoroughly understood things like constructing buildings. There are reasons it's not a slam dunk for software, but IMO there's still a lot to learn from his work.
For these you add the buffer. Sets that I scheduled were most of the time slightly faster than the schedule, until a blocker appeared which ate the buffer, but was usually resolved without eating it away completely. So at the end of a day we were always either on time or a bit ahead.
In film this is the ideal case, if tou depend on the weather or on certain locations being open, missing your slot can mean that you have to try for a whole week to get it again (or have everybody work on something else and rush them over once it works)
Some others have suggested that this might not be sustainable or lead to a sense of personal fulfillment and satisfaction to everyone. Even moreso, that might just accelerate burnout for some, which probably needs to be taken into account.
I remember single handedly shipping around 70 versions of the homepage for Apturi Covid (https://apturicovid.lv/#en), the Latvian contact tracing app, back when we thought that contact tracing would be a viable way to limit the spread of COVID enough for it to die out. I did work a lot, it was a fast paced environment where around 100 professionals in the industry were allowed to collaborate to solve a problem to the best of our abilities and it indeed did result in a positive outcome at the end, the infrastructure, mobile apps and website all being developed in record time, working successfully up until this day and being handed off to the corresponding ministry with no problems.
In contrast, our national e-Health system has been in development for years, has cost around 14.5 million euros so far and still doesn't work: https://www-lsm-lv.translate.goog/raksts/zinas/latvija/par-e...
It's actually so bad that they're considering creating a new one to replace it, something that could have probably been avoided with enough care: https://www-lsm-lv.translate.goog/raksts/zinas/zinu-analize/...
Actually, i'm pretty sure that if the software were better (not even excellent, just good), a single server rack could probably maintain the entire system (failover aside) and serve the requests for the entire country, which has just 2 million people in it. Yet, it seems that they lacked the "secret sauce" that made our own project successful instead.
Of course, as someone who's single handedly leading an enterprise transformation and doing everything from modernizing enterprise apps in a project, to improving their security, migrating over to new tech, introducing containers, service meshes, APM, additional observability etc., i don't think that time pressure is always all that you're looking for.
After all, you want those that perform highly to be able to do so sustainably.
From the perspective of the seller, these estimates don't even need to be "accurate" as long as you're making many of them and the errors are symmetrically distributed. There's even math theory behind this called Fermi estimation. The problem is, from the client perspective, they can't average out the errors over many estimates if they're only buying one thing, and from the seller perspective, I don't think our errors are symmetrically distributed. Instead, we consistently underestimate how long something will take.
It's not even unique to software. There's a huge housing boom going on where I live, with condo developments all over the place, and not a single one is actually ready to open by the season their sign out front claims they'll be open. Defense acquisition. Highway construction. Everything takes longer than budget forecasts claim.
If I had to summarise what I have read on the topic, it would be that either way works, but the up-front analysis is orders of magnitude more expensive. Some sectors require this: defense, nuclear, medical, and so on. In most areas, you don't need that and can't afford it in a competitive market.
But to hint at the complexity of the question: Not only is every project different, there are also sneaky feedback loops in there.
By the time you have completed the large up-front analysis the world is likely to have moved underneath you, invalidating some of the assumptions that went into the analysis. You can shape your analysis to account for this, for sure. But at that point does it still count as one up-front analysis or several? Is it truly up-front if it leaves options and decision points open for later?
And it's a continuum: the more options and deferred decisions in the analysis, the more it starts to look like it's clearly done in parallel with the implementation. Where do we draw the line?
There is absolutely no problem with intensive up-front analysis. In fact, this is what at least 80% of the effort should go into. But we have pretty much 0% staffing for that role. It's no wonder something doesn't work when no one's there to do it. Compare that to M&S in civil engineering.
Regarding the alleged downsides: Any wrong implementation will incur massive cost down the road (but that cost usually is not accounted for in a SaaS world), so you actually cannot afford to not do it. Assumptions can change, yes, but at least you make assumptions. In agile development, people simply ignore any assumptions and end up with a completely broken product that eats 80% or more of development time with maintenance. And of course, planning and analysis can involve prototypes and iterations.
I came to the conclusion that agile development just shifts the blame for bad planning from management to engineering. And I also think that the correct framework to planning and analysis is domain specific. The more constraints your domain enforces on your software, the better your analysis and estimation will be. Hence greenfield projects have the most problems.
To avoid estimation you need to avoid dates and avoid projects that have high timeline risks. This means you would need to judge on speed to completion, or better customer satisfaction for project execution.
[1] https://www.ametsoc.org/index.cfm/ams/about-ams/ams-statemen...
1) Experienced personnel are estimating the work (experienced technically and in the product being developed); 2) The work being estimated is of similar technology to existing product; 3) The time period being estimated is less that 6 months
If any of these are violated, e.g. new hires are given the estimation task, a years worth of work is being estimated, new technology is being estimated etc, i would not trust the estimate, and apply big 'ol bucket of risk money.
But then take the same company, even many of the same people, and put them on a new project, and the estimates go to crap.
It really takes a lot of domain-specific experience, and I mean "domain" pretty narrowly, like knowing the code base, knowing the developers, knowing the customer, knowing the external surprises and constraints you're likely to come across because you've seen them so many times before. I've never seen it work for greenfield, even in an organization with the discipline and know how to do it correctly for their older applications they've been working on for years or decades.
Example: How long will it take you to lean advanced physics? Even if I gave you a week, your estimate is likely worthless (maybe precise, but not accurate). To me this makes sense because you cannot know the unknown unknowns until you hit them (and throw in some of the journeyman fallacy)
Meanwhile, how long will it take to brush your teeth is easy to estimate. How long will it take to brush your teeth with a new electric toothbrush is also reasonable to estimate.
I like that as an example to a layman of why this might be a hard thing to estimate.
Another idiotic practice is multitasking since context switching introduces tons of overhead.
And tying it all together, the importance of a project determines its resources. A truly critical task needs 100% focus. If a manager isn't willing to assign a full-time employee to a task, then that project isn't critical.
The quest to discover and apply "laws" like this courts disaster or disruption because people are neurodiverse. ADHD people respond to deadlines in a completely different way to non ADHD - sometimes inversely, sometimes obversely, and so require different management. Same but different with autistic spectrum people, introverts, extroverts, bipolar, insomnia, sleep apneates, diabetics and long covids.
It may be tempting to dismiss neurodiversity as a fringe consideration and just assume most people are "normal", but their circumstances, both in the workplace and beyond it, may complicate Yerkes-Dodson as much as neurodiversity whilst evading detection by managers and systems.
> Time pressure is also reported to be the most frequent external cause of unhappiness among developers.
> The inverted U-shaped relationship is one typical example of this, which means that time pressure increases efficiency up to a certain point, but after that increases in time pressure cause a decrease in efficiency
A good manager knows how to sustain just the right amount of pressure.
And they know about Parkinson’s Law: https://en.m.wikipedia.org/wiki/Parkinson%27s_law
> Parkinson's law is the adage that "work expands so as to fill the time available for its completion."
Aren't these the same thing? the latter being somewhere around 24hours give or take
(there, I summarized it for you :>) ).
I may want something to be done by the end of next week and at the same time I would like somebody to spend less than 4 hours on the task. Spending entire two weeks of effort would not be acceptable in this case even if the task was delivered on time.
An example would be a task that is not on the critical path of the project but still needs to be done. There is a deadline by which it must be delivered or it becomes part of critical path and starts delaying entire project.
Assuming the task is performed by one person using the fraction of the time available before the deadline, I would like to be able to use the rest of that time (resource) as either a buffer in the project or for completion of another task. Having the person deliver the task before deadline but after two weeks of continuous work where I assume only 4 hours will be necessary means that the project was deprived of almost two weeks of buffer or there are now some other uncompleted tasks.
When we are talking about project management, there is an important, clear distinction between time and cost.
I know it is picky but, if a task "needs to be done" then it is on the critical path. Just because "build the bridge" is not needed yet, it is on the critical path even before you arrive at the waters edge.
I am of the opinion that pretty much every project management function can be replaced in software.
You are wrong.
Critical path is the longest path from start to end of the project. It dictates the length of the project. The idea is that if any of these tasks takes longer, entire project takes longer.
https://en.wikipedia.org/wiki/Critical_path_method
Has nothing to do with whether the task is important in itself. But the task on critical path is important from the point of view that it has power to affect entire project.
You can still have tasks that need to be done that are not on critical path. This usually means the task can be done in parallel of some other critical task and you have more time available to complete it than the time slot where you can work on it.
Imagine you are manager with only two employees. One employee can only write code, the other can only write documentation.
Your project requires delivering to the client completed application with documentation.
It takes 1 month to complete development and it takes 1 week to complete documentation. Moreover both employees can start immediately because they both know what needs to be done.
Which is critical path?
If the documentation task slips by one day, the project will not take more time. On the other hand if development slips it will.
From that point of view, development is on a critical path.
It does not matter which task is more important. If you had bright coder that took one week for development and slow writer that took a month to prepare fantastic documentation, the situation would revers and now it would be documentation that is on the critical path and where you will focus as a project manager to deliver it on time.
If comparing to sports, in a team, is it always the coach's fault? Or is it possible that one player is not performing as expected? Is it then the coach's issue that he/she didn't expect that?
They had hired people who knew nothing about building software (before me joining), most likely those people have never really built and shipped anything. Everyone on the management was under the impression that software is easy and everything needs to be delivered on time and if it's delayed we need to throw more money at the problem and that would solve the problem. They would continuously ask for "timelines" and never understood the word "estimate". They knew the definition of the word but could never accept the reality of the definition.
I was tasked with building a brand new core banking engine and a kyc engine. I talked to the existing tech team that was there and none had ever done anything complex. Mostly they just knew how to write some basic PHP pages to do some simple tasks like rendering a CSV table for a specific report etc... So they needed to first be trained.
To cut a long story short I was the only one on the team to be able to write productive code to build such a complex system. I hired someone who could write code to help train the existing team. However given the complexity of the problem and the tight timelines I had to solve. I basically did the design and implemented about 70% of the code myself while my other teammates were being trained. This also included building a KYC engine along the way. I remembered suffering immensely building the engine and hiring / building the team. I would focus on working with the team during the day and coded at night, since that was the only time I could get un-interrupted hours to code.
I worked til 2AM almost every night for 7 days a week for over a year (6 months to get it to launch and then refinements / continuous improvements after). To complete the engine and get it to serve hundreds of thousands of transactions every month. I was never paid for overtime. Today the company stands strong with handsome profits.
Trust me. I can tell you, the problem is not the developers / process / agile / kanban / etc..., the problem is management. It's always management. I was able to get management to understand the true scope of the task by stopping working overtime and standing my ground (when the company had recovered and was able to produce a stable income to support itself).
The overtime I did hid away all the 'complexities' and the 'debt', and prevented the company from learning. I basically ended up paying for the company's mismanagement / toxic culture with my time. Fortunately a lot of toxic people left the company, a growing company will generally push out the non-performers organically. It took a lot of effort on my part personally to resolve issues plaguing the company, establish an engineering culture, build a strong tech team that can withstand pressure and deliver continuously and can build on top of the framework I provided.
The lesson has always been. It's never the software team (at least never the team I manage) whom are not productive or doing their best. It has always been management / company culture / never truly understanding the scope / requirement enough to understand how huge of an undertaking a given project is. I can tell you without a doubt this article pretty much hit the nail on the head. I'm living proof and have the battle scars to prove it.
Now I tell management about time estimates as it is. "No fixed timeline, the team estimate, the team builds, it's done when it's done, live with it or fire me."
I've fortunately afford myself the position to be able to deliver the truth serum to management and they have no choice but to accept it. It didn't come cheap, I'm still scarred.
Management is an incredibly complex job, especially in software engineering, and the fact that we "promote" senior developers to it as if being a good developer was the only requirement to being a good manager is part of the issue in my opinion.
I'm sorry you went through this, and I'm glad you were able to extract some good out of it.
Our Management team are mostly non-software people, generally they were the problem. Not having the right education / understanding.
Everyone wants to get into tech. Not many people are well-equipped to do so.
> I talked to the existing tech team that was there and none had ever done anything complex. Mostly they just knew how to write some basic PHP pages to do some simple tasks like rendering a CSV table for a specific report etc
Where was all this money going that they were throwing at these problems if they are hiring fairly low level staff like this?
I have long learned such personal efforts are worthless, specially when the next firing round doesn't care about them, just a thanks for the fish and see you around pat in the back.