It took 12 weeks to ship an MVP I thought would take 3
boxci.dev
boxci.dev
As a mercenary engineer, I've had to answer the estimation question many times. Whenever I've given estimates, it has only ever shot me in the foot. Spolsky might have figured it out, but I haven't. So I try to avoid giving estimates as far as possible and instead focus on demonstrating velocity of a working system they can choose to stop funding at any time.
Some clients, especially from the construction industry, expect me to give them GANTT-chart style estimates down to the hour. Here is how I convey the software estimation problem to them:
If I write the same code twice, I done fucked up. So any code I write, is new code. If I can reuse code, I will, but that's not the code you're asking about. In your industry, you have built the same house many times before with the same team, on the same foundation. But you are employing me to design, architect and construct a new house that has never been built before. If it existed already, you would just buy it off the shelf instead of employing me.
So think of me as more of an architect than a labourer, and together with you as the client, we are embarking on a design that you're not quite sure you want yet, that won't collapse on a potentially unstable foundation.
I've got three tricks in my pocket that at least help me with this:
The first is that I go by the formula mentioned in "The Mythical Man-Month" — which is that the effort in larger projects is distributed as follows:
1/3 planning 1/6 coding 1/4 component test and early system test 1/4 system test, all components in hand
Developers tend to mainly estimate their coding time, which leads exactly to the effects mentioned. It also explains all the annoying situations with all the holdups once the project is "almost done", this is simply a systems test and will add another 25%.
The other is that I simply add 5-10% on top for each person involved into the project (dev, stakeholder, anything) for communication overhead. It usually holds true.
The third trick is meticuluous focus on lean delivery, which involves scoping down, early testing on as much as possible, hidden and pre-deadlines and setting tractable sub-milestones.
Unfortunately, all of these techniques don't accelerate the project, they just help you be more realistic. If you find something that accelerates complex projects, let me know ;-)
Developer experience. I've been working as a developer for a living for 18 years & I thought I was pretty hot-shit 10 years ago already, but I can probably do more today in 4 days than I used to in 4 weeks (or 4 months a few years before that).
Also regarding: > If you find something that accelerates complex projects, let me know
When possible cut scope and split the project in smaller projects with real deployments (much like you said with "The third trick is meticuluous focus on lean delivery, which involves scoping down, early testing on as much as possible, hidden and pre-deadlines and setting tractable sub-milestones.").
In nearly all my projects I now propose something like this to clients: "Ok we understand your needs and requirements. What do you think about having a V1 with only these core features deployed in a few weeks (which ideally already provide business value), and after that we iteratively improve it?". Works great for smoothly throwing away all the nice to have.
I know this was partly joking but there are two real ways:
Knowing your true critical path and proactively re-plan and manage the risk of those activities that sit on it to clear the way for it to run smoothly... that's of course assuming you planned and estimated it correctly in the first place.
If you have a real schedule that is properly resource and risk loaded, run it through a monte carlo sensitivity analysis and see if you can re-plan those highly sensitive activities or de-risk them.
An easy example is the thousands of JSON APIs you can integrate these days, where the code to do that is almost identical, but not quite. Generally the only complications are in what stupid authentication and pagination method this API came up with.
But it happens everywhere in coding. Every system might need user management, but slightttttly different each time. Or sending emails but Bob already uses Sendgrid, where traditionally you've used mailgun.
Also the software industry changes all the time. That email code you wrote ten years ago using SMTP won't be reusable. The invoicing system you built for one client using PHP and jQuery 1.7 in 2012? Not appropriate to sell to a client today.
Basically you should have written things at least very similar to what you're estimating on. Then you need to find your "multiplier". The figure you need to increase your gut estimate by to get the real estimate.
Plus, there's a lot of legal complication about reusing code with different clients, often you should be rewriting purely to be safe for them and you from a copyright perspective.
I really feel like anyone arguing otherwise is slightly delusional. I find it particularly funny the guy two up claiming frontend might be reusable code, but backend isn't.
Once you've done one invoicing/emailing/workflow/search system/time tracking/credit control/custom form designer/forecasting module/cash flow/etc./etc./etc. module for a client, you've pretty much done 'em all.
And after a certain point you can then start to estimate the complexity of other systems with a small amount of talking to the client and research that let's you know the scope and the gotchas.
if the estimate is consistently between 3x and 6x, that's what's likely missing.
Building a house is probably the most understood human activities; as you stated, you can take a standard design on a normal building lot and throw up a house in a very predictable amount of time. If you've done it enough, you can write down a schedule of when every piece of material and every trade needs to be on site.
Grand Designs always involves some combination of unusual design, unusual sites, and unusual techniques, and frequently inexperienced builders. People set out to accomplish something grand, and usually succeed, but it takes much longer and more money than expected, and especially more than constructing a conventional house on a conventional lot.
Now, besides echoing your point that many novel software projects are much more akin to a Grand Designs house than a cookie cutter ranch in a Midwestern suburb, in Grand Designs one concept that is proved true time and again is that once you are out of the ground -- once you have your foundation in, your gas and water and sewer and electric hookups -- the hard part is over and the job will proceed much more smoothly. It's hard to estimate before you get out of the ground, because the ground is filled with unknowns and unforseen problems. You might still misjudge how long it will take you to finish, particularly the finicky details that always take forever, but this is purely a failure of estimation; an experienced builder could predict the time to spackle, trim, paint, install flooring and built-ins accurately, you just didn't realize it. But under the ground, there are all sorts of problems with rocks and water tables and pipes and whatnot you can't predict.
Do we, as software engineers, have a similar inflection point in our software development? Do we ever get "out of the ground", to the point that most of the unknowns are known and we have a predictable (if still great and difficult) amount of work remaining?
Estimating should be fun, plus there are various ways, such as estimating as a team (majority wins), t-shirt sizes, etc
It's always maddening to get into debates about this stuff with project managers who believe this nonsense. I know the Fibonacci point scale is supposed to address this by giving PMs something, but I've only ever seen it turn into time-estimates-by-proxy.
These days, I only ever give estimates in terms of time scale. A task will take "hours", "days", "weeks", or "months". As long as it's within my power, my team will only ever estimate on that scale.
The only question I ask around prioritization is "Is it worth doing?" If the answer is yes, I ask the stakeholder if they care how long it takes. If they do, I ask for a range and give a best estimate if it's achievable. If the answer is no, I ask why they care how long it takes if it's the most important thing to do.
My engineers know that they're expected to do that one project to completion, and we loop in other stakeholders (marketing, sales, QA, etc) progressively as we approach completion. Admittedly I work at a company of 20, but it works remarkably well, despite the absence of a "schedule"
We also work with a Kanban-like methodology. Basically a pull system with a hard limit on WIP.
Well... would you rather get $150,000 two years from now, or $90,000 next month?
You generally don't want to be making any decisions off ordinal comparisons.
Ask them to prioritize the tickets. Maybe they want some rough estimate to decide between some of them, maybe not (which is, suprisingly, in most case). During the same meeting you can groom the tickets.
The developers then start working on those tasks, when they are ready you release them according to your company policy and calendar.
By then everyone have forgotten about Scrum, management is amazed you release so much stuff, you can probably put all that in a one hour max weekly team meeting and everyone is happy.
I'm pretty sure that's applicable in a lot of pure player tech teams and quite scalable (I managed up to a 15 devs team this way but PR validation etc overtook the management part a bit much).
"Why can't we better estimate these things?" Because when I give honest estimates I get threatened by my boss. Your process is not reality driven.
"Can we break this work down into smaller stories?" No, I can't break down the final integration of all the parts into a smaller story. Or more commonly, they told me we have "too many stories" and can't keep track of them all when we break them apart.
The best is when a Product Owner or other business type says "I'm not a technical person." Then why the hell are you working in a such technical business and calling the shots on stuff you don't even understand?
Same people: you must complete assignments in the estimated time, or else, be prepared to work overtime including weekends.
It drives me bonkers.
I ask first will the task take 20 years to finish and they usually laugh and say of course not. Then I ask what about 20 seconds and they laugh again. We keep going on both sides of the boundary until they stop laughing and it starts sounding more reasonable.
Then I have something to work with like 3 to 6 months which gives me more options. I can accept the risk or I can take other steps to reduce it or break it into more manageable bites. It’s not perfect but it’s worked pretty well.
Another approach I had some success with was estimating the 80% likelihood case, I.e. I’m 80% sure I can finish this in the time I’m estimating. Less accurate, but much more predictably on schedule, which is more useful for many stakeholders.
A last one is giving rough estimates and being clear about how rough they are and how much time it will take to get more confidence. I was handed a 50 page PDF of API docs for a service and asked what the estimate was for integration. After 15 minutes I said “1-4 weeks”, if you want more accuracy then I’ll need a few days. The answer was “no problem, we wouldn’t consider it unless it was < 1 week”.
These examples are all on the weeks to low number of months scale, but the same applies further up. Having a good understanding of the codebase and domain are very useful.
Estimating isn’t easy, but treat it like any skill, it can be improved. It requires some flexibility from product managers, but open communication goes a really long way and results in better decisions overall, less wasted effort.
I don't have a statistic, but I'd be willing to bet the majority of code in existence is not algorithmic in nature.
Every iteration there'd be one or two small tasks that blew up to >1000% their original estimate.
If everyone could do that trivially, then there would never be any problems.
The problem is that on a non-trivial project, these breakdowns are guesses which carry considerable uncertainty. And the less you know, the more the unknown-unknowns get you. If you write a detailed breakdown of a major piece of software before writing a line of code, at the end of the project you will look back and laugh and laugh at your own innocence.
As just one random top-of-mind example, consider Carmack writing Doom's rendering engine[2]. He tried several approaches that didn't work before striking the right balance with pre-computed BSPs. Some of the things he tried were fundamentally good ideas and would later be used for different games - but didn't work on the hardware of the times. How do you estimate something like that? "I'm going to spend a month on my first approach. That could be one month or two. If that doesn't work, (50% chance), I'll read research papers and textbooks (1 week - 3 months) until I find a promising technique. Then I try the most promising of these; since this is an approach I haven't even read about yet, it could be one week to implement, or six months. And there's some chance that will fail to and I'll have to do it again." The final estimate is anywhere from one month to a year. You just can't know.
[1]: https://www.amazon.com/Software-Estimation-Demystifying-Deve...
The only thing I can do is move them to the front of the project as much as possible, so we have the risky bits as early as possible. Once we get them to work the rest should be smooth sailing, but I'd rather fail when that time wasn't spent yet. PM agrees.
But the way you go into a market is by figuring out what doesn't work. What is an unaddressed pain point. This is a hard thing to do by definition (that's why the payout of being successful at it is so big), but if you manage to do that, you don't need 10-15 persons for an MVP... in fact I'd argue you actually can't truly do an MVP if you have a 15-persons team. You need the large team to expand the MVP, but to identify it/ have a customer-worthy demo, the 15 people will just get in your way.
The more likely explanation is that your "strong differentiator feature" is not really as strong as you'd wish it was.
No it doesn't imply this. One granular piece of Waymo's software is "develop a component that understands the external environment as well or better than a human". It's easy to say that, and it's even relatively easy to break that down into risk scenarios etc.
But developing it is.. non trivial. And there are parts in it which it was unclear if they were possible.
But that is because large parts of Waymo were a research project, not software engineering.
Regardig the "research" vs "engineering", I don't think you can cleanly separate the two; many otherwise straightforward projects include a research component, even if it is just about using a new browser feature in a novel way, or some such.
Obviously, you have implicit constraints in mind. "Is something possible given $constraints?". And the set of those constraints is where the line between research and engineering starts to blur. If you set $constraints to "in time and under budget for the project", which seems like a very reasonable value, then it turns out that a lot of programming tasks become research work, unless you're only churning out cookie-cutter websites.
And no, "in time and budget" doesn't suddenly make something into a research project.
Speeding up some technical process might be research (eg, the old 1TB sorting benchmarks drove research).
But project constraints, and the fact that the team themself has never used a specific technology or whatever doesn't turn it into research. Yes, I know that's a term people use, but it's a different thing to scientific research.
E.g. I worked on a warehouse app before. None of us ever had before.
If you have no experience with any of this, how do you break down picking: SKU design (ours weren't just random numbers), barcode printers, and barcode scanners to be used in your software project? That part alone took us about 7 weeks. Including a complete SKU redesign because our initial test hardware worked better than some of the stuff we got later.
Nevermind all the hardware integrations, which was also new to us: multiple printers (barcode label, shipping label, pick list, packing slip), barcode scanner, a scale, all seamlessly integrated into the warehouse application.
This also reminds me, one of the services we decided to use had a serious (for us) bug that we didn't find out about until late in the game. They didn't fix it for 3 months and it blocked us for a while. So yeah, you could be very well end up integrating more than once. You can't really break that down into trivial parts.
The trick is to treat everything that you can't break down to a triviality as an area of research until you've solved that problem.
So, say, you plan three days in a sprint to research a certain requirement, the results being a set of small east-to-estimate features and maybe more tricky features.
If you're working in an area where your team have absolutely 0 experience then there's no way you can estimate anything accurately. In these cases I hope that you're working with a small team (2-3 developers and a BA/PO) who are highly experienced and work well together. Then you should run flexibly with features. Work Kanban style and implement the 80/20 effort/win features - and don't be scared to drop a feature if it's looking like it's not part of the 20% effort group of features.
If you can't break it down you don't understand it and if you don't understand it you aren't ready to estimate it.
now the hard part is selling the idea to the management, like "here are our ideal estimates, lets multiply it by 4". when doing freelance work i usually bill for the ideal estimated time multiplied by 2, the rest being my risk, on the other hand any changes in the requirements, or miscommunication is a risk of the client.
I see this over and over again with people I know bootstrapping projects - the over optimism I think is just part of being a builder. Sometimes in startup circles there seems to be this meme of how you should 'throw together a prototype of the tech in a week' being the standard way to get going with an idea. In practice though, seems so impossible that you could get anything even useful shaped built in that amount of time. For me when I get down to work on something, even just plumbing together the vague structure of an app takes longer than a week full time.
Most software projects are more like R&D than construction. For the adventurous analogy, software is like hiking an unknown trail. You might generally know where you're going, but you can't foresee the river, detours, and feral hogs you'll have to battle to reach the end.
You simply can't estimate software the way you can more tangible products. You can do Scrum planning poker and pretend your're not playing Numberwang, if that makes you feel more in control.
Then you look at big civil engineering projects that seem to always be over-time and over-budget.
The treatment of estimates in project management for software is way more like the assumption we have an assembly line where even if there are issues these are routine enough that they can be padded in. The problem is that software is an exploratory process and not an assembly line.
It's even worse the more creative the project is. I've worked in games for fifteen years and I don't think it's possible to estimate when a game will be 'good'. Making one is a process that requires a lot of feedback, iteration and scope management. It's part of why we see so much reflexive crunch as the reality of making a game hits poor project management strategy based on this assembly line thinking.
The words Gantt chart make me shudder.
I think those projects might be best compared to the software projects we think of here...
Discipline and time estimates are important, but not more important than a product setup for success. One that is easy to sell and market because it has value, is polished and a good experience.
Game development notoriously is late, and as long as the developers have time to finish it right people forget. Everyone knows when a game company and the business/marketing/financial side pushes out an incomplete or buggy product, it can permanently harm perception.
Valve Time is something that is really common game development where engineers/developers/designers/product people are still in charge and making good solid product. [1] Technology, innovation and especially game development needs time to make it fun, a good experience and a solid product.
Creativity can't always be rushed, there must be "open" and "closed" modes of development, not crunched in "closed" only modes. John Cleese has an excellent talk on managing creativity and I recommend all people I work with watch it, very valuable insight. [2]
I'm not off by more than 20%.
Consider that many software projects led by experienced software engineers go so far over time that they are never finished.
Category 1 is
>I've written a script to get the data from tables 1 to 24 of this database, now I need to write one for table 25. It should take me an hour.
>Oh look, it took me nearly 2 hours because I forgot that table is owned by a different user to all the other ones and I didn't have the right permissions.
But this (a) is still quite optimistic, and (b) ignores variance. The potential time a new research project could take is unbounded.
Whereas padding in a more assembly line like situation can actually be measured and forecast from historical data.
Also, I think by definition, a non-trivial project obviously hasn't been done by you many times before. Doing something you've done a dozen times is trivial in this sense.
The problem is that most people I've met dont really understand what they are actually doing and/or have the leadership to grow people to achieve.
It is not trivial in the least and requires a great deal of lead time to even understand the problem(s). This field is at its infancy, and I dont yet know if my insights can be replicated yet.
I'm thinking of writing a book where I illustrate thought exercises and career challenges that help people grow from junior to senior principal, but it is proving difficult.
How long was the discovery and design document phase on the 2 year project?
Some sketch of the end-design is always important, thinking about major components and future evolution, but a detailed plan has never been worth it in my line of work.
The most generous way to view what you described is that you’ve had to cancel your project half way through and start a new one. There’s nothing you can do to make business decisions like that work smoothly. In my experience though, a more likely cause of what you’ve described, is that the project was just planned poorly to begin with.
The kind of problem we have when launching a new product is that we can have a well-defined list of everything that is required of the new product to be assured of successf - it's going to take 5 years to build that, with at least 20 people.
What can we cut and still have a successful product, that we can launch in say 1 year (or 6 months) with say 10 people? That is much harder to say, and different product leaders have different opinions, based on discussions with different clients. And the consensus opinion may simply change.
So sure, the architecture is generally driven towards that 5 year product idea, but the short term estimates are always going to be geared towards a specific subset of that. And that subset is easily subject to change, from one quarter to another. We won't throw away the design if that happens, but we will definitely throw away any longer-term estimates.
And charting out a 5-year plan that we would only 're-jig' as priorities change would be a recipe for certain failure - teams change, people leave, etc. You can't hope to have an accurate estimate on that time horizon.
> a well-defined list of everything that is required of the new product to be assured of success
Only bad software tries to be everything to everybody (not to shit on bad software, it’s a multi-billion dollar industry). Your customers have a problem that other products weren’t solving, your solution to this problem is the core of your value proposition. It’s the only thing you need to implement to launch your product, everything else can be added iteratively over time. If you don’t know what that problem is, then you’ve failed before you’ve even started. You’re not driving your company forward by writing code as soon as possible, you’re just doing skids in the parking lot.
I’ve seen so many people get sucked into the fiction that their product is by necessity too complex to plan and design properly. In reality I’ve never seen it to be anything other than a cover for not actually knowing how to run a product well. You’re not saving any time by making it up as you go. All of the design decisions you need to make still need to be made, you can choose to make them in a controlled manner, or just do it randomly and hope you pull through. But good design is not an emergent outcome of the resources you devote to implementation.
Most work can fit within quarters and get executed quickly, but it takes something else to have long term vision.
The discovery process was paying attention and looking at what is actually the goal, and it generally requires the discipline to slow down and focus on the future without giving into the short term time sinks.
I had the benefit of 70-100 minutes per day of being stuck on a boat commuting for years to focus on discovery. The particular project that had a two year vision had about six months of shit to eat in office while thinking and planning on the commuter boat. It is amazing what one can accomplish by simply writing and the time to focus.
(If I seem bitter, I'm not. Just sharing my lived experience.)
There are many other challenges as you point out.
I am currently on year two of a five year project. The key (and part of the essential difficulty) is finding quarterly or six month valuable deliverables that help ease anxiety of everyone involved.
It is not easy, but it is possible.
While all this is going on, I'm also working on a design document for a potential 10 year project. I doubt that I will pull this one off since I'm having difficulty finding those quarterly deliverables, but I'll keep grinding away.
On the other hand, for a complex SaaS back end and front end doing something unique, a complex system to which you are being asked to add something groundbreaking, where you need to pay off various tech debt as you go in order to get the job done, where you are negotiating the requirements as you go along, and finding dead ends and impossible things that you can't do that you thought were simple and vice versa - I think it is hard to estimate. I think in that case estimation is a waste of time and it should be thought of like a "value train". If there is a genuine deadline (rocket launch) then best to manage it well with appropriate large buffers and dependency management. I don't see this being done anywhere, it is usually the business trying to haggle down the engineers on their estimates (even though the engineers are paid a salary so ???).
I’d guess that freelance developers who are paid per project, and thus have to own their P&L, are very good at estimating.
The rule of thumb i use - anything which takes more than 3 days needs to be broken down further into tasks. If any task takes more than 3 days during execution then it needs to be broken down to more tasks.
Whenever I’m asked to estimate I tell them that I’ll take a week to estimate a 4 week piece of work they almost always say don’t bother - then when the made up estimates are wrong we can discuss the fact that they didn’t bother estimating.
It is a funny thing but works every time (in corps).
I think the real reason estimations are hard is that basically every developer is working on projects too hard for them. Like, if you are so good at your current job that you can reliably churn out good solutions with ease then instead of just producing you would move on to do harder things for better pay.
I was talking with my project manager today, and he was asking me if we could finish on time if we got twice as much time as I estimated. My reply was that we’ve tried that 4 times now, and we’ve never reached the point where we actually finish within the time estimated.
So increasing the time would just mean we fail later, not that we suddenly succeed. I’m firmly starting to believe that work expands to fill the time allocated to it.
- The tool you planned to use has a bug/defect that blocks you.
- The people who said they could give you some information you need, can't.
- You misunderstood or were misled as to the capabilities of a tool you need.
- The parts you need don't actually fit together as planned.
- The documentation you were relying on is wrong.
- A delivery you are relying on won't actually be ready on time.
- A process that's worked every time has an unhandled edge case that you're going to trigger in this project.
- A regression in a software package will block you because you can't use the previous version for other reasons.
The more moving parts there are, the more likely it is that one of the interface points between them will block or delay you in one of these ways, potentially for weeks, maybe even months.
So for any nontrivial project, you should expect a 2x-8x delivery inflation over your "safely padded" estimate. You have a 75% chance that you'll deliver within 2x of your padded estimate if you've made a good estimation, sliding to worse as the number of moving parts increases.
Developer to development lead: "It'll take about two weeks"
Development lead to project manager: "It'll take about 4 weeks"
Project manager to self: "It'll take double that plus 2 weeks"
Project manager to management: "It'll take 12 weeks"
Management to client: "It'll take 8 weeks"
Actual time taken: 16 weeks.
All software estimation boils down to the long established scientific methodology:
(2 X what the last person said) optionally plus 2 weeks
???
Doesn't seem consistent
It's consistently correct.
In the consulting world there is the rule of three, where you multiply everything by three, not just how long it will take.
Then he'd say, "okay, what would it take to do what you just said in T/2?" Then you'd cut stuff until you got to T/2. The project then, very often, would take me the original T.
This seemed to work a lot better than just doubling the initial estimate to 2T.
Another estimation trick that I've found to be very effective over the years is to start with the same sort of task breakdown and finger-in-the-air estimates. Next, start to make a mental list of all the things that could possibly go wrong with each of those tasks. Add those to the list. Most developers with some experience don't add these initially because they don't consider them to be "tasks", but can come up with a huge list surprisingly easily when prompted. Now factor in all the extra time for dealing with those unexpected-but-will-definitely-happen issues.
The original estimate is your best-case, since almost all developers tend to estimate optimistically. The new one is the worst-case.
A good rule of thumb is that the worst-case is 16x the best case, so if you got less than that, it's a good indication you didn't think of enough things that can go wrong. If it does follow the typical curve, 4x the best-case estimate will be a good expected-case (75% chance of success).
What this manager did right was prioritising the tasks, because estimates are just the starting point. The next thing is to know what absolutely must be done for MVP and what is nice-to-have.
If you have 3 weeks to launch an MVP then that MVP needs to be achievable in about 4 days. Otherwise you are setting unrealistic expectations. If you actually finish it in 4 days (unlikely) then well done. You can start adding polish in order of priority, starting with a load test in production to find out as many "unknown unknowns" as possible.
This is one of my favorite tools for development. I always sketch out an idea (no coding) and then try to describe it to someone with zero CS background before diving into an mvp. The times when I didn't follow this process - well - just as op described while trying to document the cli, I basically wrote the damn thing 2-3x over. So instead of a couple hours of chatting with other people, I had to spend days/weeks running in circles around my own work. Please, if you have juniors or if you are a junior, try to drive this lesson in.
The static blog pages though are all cached via nginx so I was surprised I needed to increase the nodes & relicas. Short answer is I don't know, and will need to investigate, but I suspect it's because the kubernetes node instances themselves are fairly low powered, there's a low cpu limit on nginx, and perhaps nginx is doing a lot of work serving the js bundle to so many simultaneous users, which really should be hosted on a CDN (probably also the blog).
Probably shouldn't. If you are already managing an automated environment, creating another environment that is easier to manage has nearly no upside.
Unless that phrase was meant to use the CDN you talk about later, but again, do you need a CDN to solve the easiest part of your infrastructure? (Or rather, do you have enough traffic so that it isn't the easiest part?)
I'm guessing he likes to tinker.
"Think of me more like being a journalist. You want a good story and have a rough idea, so I will find the right people and interview them (domain experts, business people), try to figure out how to distill their knowledge into something that is both accurate, yet not too detailed and most importantly has to fulfill the needs of my readers/users (easy to read, yet super insightful, with some pretty images etc.). Wheter the article is about nuclear physics or siamese koalas is secondary, the process is more or less the same, yet I am neither a koala expert nor a nuclear scientist. You are the expert."
I also try to explain why I am unsure about estimating even relatively small tasks like this:
"You know restaurant (or another famous location) Y, right? We both know how to walk, I mean we have had like 30 years of experience in doing that, correct? So how many minutes does it take you to walk from here to restaurant Y?". The more people in the group, the more interesting it might get. Estimates usually differ by a factor of 2-4. People usually cannot even correctly estimate a trivial thing like taking a walk.
The reason it's an underestimate even when I know the distances and usual times is because I think about the ideal set of conditions (not missing train connections, clear streets for fast walking, being able to find the place we're meeting instantly on arrival rather than looking around for the entrance for 5 mins) and go with that. I never account for the possibility of missing a train connection by 15 seconds and then having to delay the journey by 30 mins. Even though I know, of course, there is a non trivial chance of that happening.
i.e. if you estimate 7h, then calculate 2x7+1 = 15h.
If you estimate 1d, then calculate 2x1+1 = 3d = 21h.
It's a silly joke, but it also gives insight about how we perceive time and the elasticity of estimates.
And of course, instead of multiplying by 2, you multiply by a time factor that depends on the risk factors involved (which are also guesstimates, but for me 'x2' would mean a rather low risk factor).
- Technical uncertainties (e.g. self driving cars)
- Human uncertainties (multiple different teams building single large software system)
- Scope uncertainties (we don't know what we are actually building until we get into the weeds)
Outside of these we should be able to make reasonably accurate (+/- 30%) estimates. I wrote a blog [1] about it if someone is interested but here are the main takeaways,
- Don't just estimate writing code but also include time required for testing, documentation, communication, setting up infra/deployment
- If certain part is hazy ("is there a reliable python library for speech to text") then research it enough to know the path ahead
- Breakdown the system into smaller units until you feel confident to estimate each piece
[1] https://blog.amirathi.com/2018/02/05/science-of-software-est...
Technical uncertainty is also more common than not, unless you are using a dying platform, it will change between the estimation and implementation.
Human uncertainties are an avoidable one, if you work alone. If you are in a team, they are also certain.
Or, to put it shortly, yes it's perfectly possible if you are doing a university project alone in an old platform without an active community.
I once gave a 3 days estimate for a client project. 2 days to make sure I build it right and a third for testing. It turned into a 7 weeks project: https://idiallo.com/blog/18000-dollars-static-web-page
My estimates get accurate once I know where we are on the spectrum from quick and dirty to pushing the boundaries of what's possible. There's a 3 month version and a 3 year version of a lot of ideas.
Each project is six weeks. You deliver the best possible version within those six weeks.
I'm always confounded when managers want a time estimate but cannot provide a value estimate.
The goal is the same: fix important/urgent bugs, develop product and keep good close velocity.
> Then I stopped, took a step back and looked at the product from the perspective of someone who would want to pay for it.
Though I think this is dangerous advise. You do not know what this perspective is like (unless maybe when your product is a “scratching your own itch”-type product) and therefor making design changes makes no sense. You will never know if users maybe even preferred your first “design”, or that there is a different blocker like landing-page, installation-instructions or anything really preventing them from starting using it.
Totally agree with what you're saying here and let's see, I might write another post in a few months essentially saying this was a mistake and a waste of time, but yeah if you're interested in seeing a comparison, check out the twitter account for the blog @zero_startup and you can see the old screenshots and the newer ones.
When can you say with fairly good certainty how much longer it's likely to take? In my experience it's about 1/3 the way through the project. I'll give you my reasoning behind this number.
There is a model of software defect discovery called Littlewood's model. It basically says that the rate of software defect discovery is a random variable. As you discover more defects in the code, the number of defects that are left to discover diminish. Assuming you put in a constant effort to discover defects, your rate of discovery decreases. You can estimate the number of defects left in the software, by looking at the decrease in discovery rates over time. There are lots of scholarly articles on Littlewood's model, so I won't go into more detail than that. There are newer models too, but I always found Littewood's "good enough" for my purposes.
It occurred to me that requirement discovery might follow a similar curve. As time goes on and you understand more and more about what you need, the number of things to discover decreases. If you work on your project in a constant manner, the rate at which you discover new requirements will decrease over time. I took data for a number of project and the discovery rate curves were very similar to defect discovery rate curves.
The key is to look at the rate of change of discovery. Once it is tailing off, you can estimate the speed of the drop and get a good idea of how much work you have left to do. Each curve is different depending on a number of factors, but by the time you are about 1/3 the way through the curve, you have enough information to estimate the parameters.
Note: Littlewood's model doesn't actually hit zero, but you can set a threshold that is "close enough to zero". Where you place it will change the 1/3 figure, but I hope what I'm saying is understandable.
If I was asked to give some hints, I would say:
- in optimistic and pessimistic cases don't fear to use extreme values
- don't estimate on your own, take reasonable amount of developers with variety of experiences
- don't give exact value to client, give probabilities ("it is very likely that we will be deliver this in between X and Y man-hours/days)
All estimates I made using PERT were quite accurate (<5% time difference between estimated and real execution)
From my experiences, I've found that 3x my initial "estimate / goal / ego trip timeline" gives the most realistic picture. Sometimes I can hit my best; most often things come up. It took a little while to accept, mostly because I wanted to do my best always and be able to promise and fulfill it. But Reality hit back, and I had to adjust.
It's hard to see the network effects of each micro-interaction in long range projecting, but once you really break down how long each thing could take, how that would affect each other related part, and who/what else is involved at each step/layer, it clears up the fog.
<gulp>
I'll probably give everyone an upvote here as a) it's nice to see I'm not alone, and b) there's some really good tips.
Not sure why every tech interview introduces a whiteboard with an algorithm problem when all they have to do is ask me "how long will this simple command line client take to build" to really see the difference between the way I attack a difficult problem!
The idea is to learn as much as possible (read The Lean Startup if you haven't). If you build too much in one go, you're going to have tied yourself to one vision of how your users will use the product and be more closed to learning from them what they actually need. Note that you were persuaded by a designer to make the design nicer, not a user, potential user or even product manager.
What was the problem with launching, and then deciding that design was the next priority and working on it while your product was running?
[disclaimer: I definitely haven't got this figured out as we shutdown the company and I'm back building Big Expensive Systems]
Great question, and one I asked myself when I was about 2 weeks into the redesign :-)
On reflection, basically no, you're right, there was no reason I couldn't have shipped and then improved the design in the meantime. The thing about it not being good enough for people to pay for was just my opinion - not tested with real customers.
The viable I had in mind was 'it works' - later realised that actually viable was 'I think there is a chance I can ask people to pay for this' - turned out that second part around doubled the already overrun build time!
Estimating requires you already understand the very thing you're estimating. This is why plumbers, and electricians give you reasonable estimates. Heck when we hired movers after buying our first home they estimated two hours, and were done just a few minutes before the 2 hour mark on the spot.
The issue with programming is, your tools change, your approach changes with new iterations (usually for the better!) and you notice new problems, or sometimes you do one thing in 5 minutes, and your second run through using a similar approach is broken because it requires an edge case you didn't anticipate code wise.
It's doable, really, and it feels an awful lot like waterfall when you're in the moment. The most important steps were:
1. Write it down 2. Don't keep it to yourself
Everyone else is smarter than me and provided immeasurably good feedback.
I shared my deadlines and progress on this build via the blog and on twitter under @zero_startup and I would say doing that probably pushed me to ship the MVP at least 50% faster.
Marketing advice: define "CI" near the top of the first page. The word "Integration" is not on the page either. I am familiar with Continuous Integration but not the abbreviation CI.
Adding to the confusion, the font on (most of) the home page is sans serif, so CI cee-eye looks like Cl cee-ell.
Moreover, you mention CLI on the page... though I did know what that is!
Good luck with it.
> Adding to the confusion, the font on (most of) the home page is sans serif
I mean, except the logo at the top of every page.
>Seems easy right - it's a CLI - just document every option and you're done in half a day. That's what I thought too.
>But then you realise that each individual option is its own thing that requires thought to explain, and crucially, interacts with all other options. It's not enough to explain it in isolation. You need to tie everything together. You need examples. You realise from doing all this that there's a simpler way to name or combine options, you go back and make that change and have to update all the examples, explanations, etc.
You question whether your MVP at that stage was viable, but when I read this, I wondered whether it was actually minimum. Is it possible that you added too many options and your MVP would have been easier to ship if it had been more limited in scope?
>The product did not look like something anyone would want to pay for
One pitfall to look out for is that you probably can't accurately predict what customers pay for until you talk to a real customer. Maybe things like aesthetics that seem important wouldn't actually be important to a customer if your product solves their problem. A designer gave you feedback, and they no doubt have valid expertise, but they're also not your customer.
I had the experience of shipping too late last year:
https://mtlynch.io/shipping-too-late/
tl;dr of my takeaways:
* Even if you've been warned about startup pitfalls a million times, sometimes you just have to make the same mistake to learn it.
* The goal of the MVP should be to get something in the customer's hands ASAP so you can hear their feedback. If you focus on polish and design, you might be polishing something that nobody wants.
* Question whether your "must haves" are actually just ways of delaying scary conversations with customers.
On the 'get it in front of a customer' thing - I agree and struggled with this decision a bit after I'd made it as a result. Probably the best way to validate you're building the wrong thing, or it's not ready yet, is hearing 'no' and hopefully why.
Super interesting factor in this case though is that basically I am the customer - it's really a product I'm building because I want it to exist. I asked myself the question, would I honestly pay for this right now? The answer was no. So at least for customer number one, me, I did kind of have that answer. Maybe cheating a bit though :-)
It's a bit dated now, but most of the advice is still relevant to solo engineers looking to turn solo founders.
My notes: https://mtlynch.io/book-reports/start-small-stay-small/
Very cool to hear from other bootstrappers / solo founders like yourself talking about things they've done, successes yes but also failures and lessons, and that was also my aim with this post. I feel like it's a pretty small niche within the much larger overall 'startup' world.
Consider instead shipping stuff incrementally, feature by feature, when features are done, without any hard deadlines. Both the speed of delivery and quality will be improved compared to hard deadline driven development.
Plans are worthless. Planning is indispensable.
You can't plan for a product to be viable. You first build a product and then you verify whether it is viable. At best you can formulate a hypothesis about what set of features you think is good enough in terms of KPIs and then plan to build only that. However, you have to factor in the likelihood that your hypothesis is wrong and also that the process of building something should result in refining that hypothesis over time. If that doesn't happen, you are not learning and you are probably not really building a viable thing.
The fastest way to get to viability is to take baby steps: short sprints/iterations, ship often, re-assess where you are every step of the way. Do the most valuable/risky/uncertain things as early as you can so you can adjust course if your assumptions about their value turn out wrong. Most startups get this wrong and fail for this reason because by the time they figure out they are on the wrong track they've already wasted most of their seed funding on building pointless things.
The lean movement tends to focus on the M part too much which has a built in risk for products to be unexciting and ultimately non viable. It's great if you are copying somebody else's business model or building some kind of market place. It's not so great if you are trying to do something new.
Lean has a tendency to postpone value creation until you've built a lot of low value commonalities like a login system, user management or crap like that that every startup seems to spend ages on without getting it really right. If you hear the words MVP and Android IOS and web in one sentence that translates as we're building a lot of common functionality three times and all our experiments have 3x the cost. The chances of that being utterly unremarkable and non viable are huge.
The minimal thing would be to postpone that stuff until you have something worth logging into and worth having multiple implementations of on multiple platforms. Building a good mobile experience is a huge investment. Don't even think about it until you have something viable.
You are not proving the viability of a login system with your MVP nor are you proving the viability of a slick IOS experience. Your thing is viable if despite obvious UX and feature issues you still get positive KPIs. Once you get there, you can justify the expense of making it better. So spend as little time as you can on stuff like that instead of making it the top priority in your first iterations.
As long as you are doing minimal things you are not actually creating a lot of value. You are actually postponing value creation and viability. It's the hardest things that create the most value and that are the hardest to plan and the easiest to postpone when planning. Therefore I believe, Scrum is the wrong process for building something that has a high risk/reward balance and the right process for building something that has low risk/rewards. All the value is in the stories that everybody struggles to estimate. Scrum results in most resources getting sucked up by the least valuable stuff.
I agree with the author in terms of estimates and their assessment of the process, but the point I'd like to make is that the vast majority of those 12 weeks it took them to launch the MVP weren't spent on core, differentiating functionality, but rather on boilerplate that's common to every SaaS product.
Important but non-core features like billing, accounts, documentation, and a marketing site end up causing many projects like this to fail before they even get launched.
If you can launch MVPs like this significantly quicker albeit in a more constrained yet platform-agnostic way (e.g., using serverless functions as an abstraction), it becomes really powerful for makers like the author of this article to focus on core value and ship quickly instead of wasting time on all of this other stuff.