Hidden costs of constantly shipping new things
mindtheproduct.com
mindtheproduct.com
1. A bias to ship and a bias to ship new things are not one and the same. A lot of the problems, such as a failure to iterate on existing products/feature, sound very much a product of the later, not the former. If anything, the issue "Insufficient Iteration" is probably not correctable without a bias to shipping.
2. A bias to ship and a bias to ship things that impact your customers are not one and the same. There is a note about an early shift to micro-services. I can't speak to this organization, but, generally speaking, spending time on internal engineering work to the detriment obvious missing features is a common issue with early stage companies.
3. A bias to ship and a bias to ship to the right customers. Specifically in regard to the high value, high demand customers.
4. A bias to ship and... idk what to call this? "The product specs were well thought through, sometimes crafted for months." TBH, the problem with this one feels like a lack of a bias to ship.
As described, I think the real culprit was a lack of or poor prioritization. FWIW, I suspect the author and I may actually be in violent agreement as I did find my self nodding with most of his lessons learned. Though I'd be careful about letting too much hindsight bleed in (eg - do situations that'd be improved by more decision documentation justify the effort of documenting all product decisions, especially in the early phase when the product is rapidly evolving)?
The ship it constantly is probably ok when you're small, but at some point you need to become more mature than that, both as an organization, and as an approach to the software itself.
One reason for this was how heavily we prioritized first mover advantage. We pushed extremely hard to get into new markets, and while strategically this worked out extremely well for us, it meant that later development was hamstrung by earlier decisions. Any iteration on old features almost always meant rebuilding it entirely, and we've only just started to get to a place where previous efforts to do this are now paying off in increased velocity.
I think there's an argument that this could be good, since you're only improving features when there's some other value you can ship. However, I think it's short-sighted, because it means prioritizing small iterations has an unnecessarily large cost, simply because that cost wasn't paid down slowly over time. While I do think prioritizing new markets early on was a good call, that mindset bit us in the ass in a big way when we tried to ship a very flashy new feature later in our growth cycle that crashed and burned because of tech debt, and it took us over a year to fix everything to release it. If we had taken a mindset of building for the long term five years ago, which was already years after we'd gotten to a comfortable place in revenue and marketshare, we would probably have paid off most of the major tech debt by now.
I honestly don't know how a company can get past the addiction to shipping, at least not during its growth period. It can be strategically necessary, and I imagine in only a two-year period at the early stages of a company, like what's in this article, it's impossible to avoid. But leaders must have in the back of their heads the idea that they will need to start tipping the scales toward standardization, building to last, and building infrastructure as well as features sooner than they'd probably like to avoid larger problems in the future. And I suspect the only way leaders will really take that seriously is if they've gone through it before.
EDIT: Clarified a few points.
It's hard to change culture; you tend to keep doing what worked before even if it's not optimal anymore. Yesterday's adaptive behavior is today's unfortunate practice.
In fact, once we started investing in that kind of development, we did do that with a successful new feature that we developed much more quickly than we could have in the past, and it's because of that success that leaders are finally starting to prioritize building standardized and reusable infrastructure.
EDIT: I think another way to say this is, once we knew our strengths and how we'd be successful in the future, we should have operationalized that and hardened it, so we'd have a sustainable advantage over competitors. Instead, we kept the "move fast, be flexible, and throw man-hours at problems" mentality too long, and it put us at risk.
In the long-term, I do think product-wide you always have teams at different levels of maturity. Some know more about the problems they're solving and how best to solve them, others know less. Some will move fast, some will build to last. The more a team knows about their problem, or the more a problem cuts across teams, the more I think they should build to last, so investment in those areas scales over a large group or important areas of the business.
Paying back the debt, he said, usually requires too much knowledge of a system to cycle teams in and out consistently enough to ensure everyone gets a piece of the new versus the old to keep them engaged.
Whether or not that was backed up by hard data, I'm not sure. It was an interesting angle I hadn't previously thought of though.
Very true. The good performers will be “rewarded” doing the things that are really important: keeping things going. And then you have low performers and interns doing the new stuff.
As someone who found themself in the "holding the fort" team, I can confirm that this analysis is at least anecdotally true, although the terms I preferred at the time for the two teams were Morlocks and Eloi.
There’s not an easy fix though, as most of the time I see orgs just punt at some point and allow the dev shop to write off all their tech debt by reassigning responsibility to an ops team.
While that's probably true, it's a difficult balance to know when exactly you should start moving back towards the better development practices. Too soon and you die, too late and you can no longer move because you're too inefficient.
This is a false dichotomy, and there is plenty of research (and experience) that shows shipping higher quality software actually takes less time and money.
My experience would suggest shipping higher quality software is a lot cheaper in the long run, but has up-front costs that often times startups can't afford.
They could have shipped higher quality software quicker by hiring 2 senior developers for similar cost.
Slapping it together pays off today but at the cost of acquiring a debt that costs lower average velocity and lower agility until its repaid.
A lot of small businesses make the decision to take on debt to move quickly at the beginning because the most important thing is initial growth and proving that you can actually make money.
I believe you when the business knows exactly what it's building. In the context of the article and the comment I was responding to, building high quality software wouldn't have helped them early on, thats kinda the whole point of tech debt and the discussion here. Knowing when is the right time to start changing the engineering culture and how easy (or hard) it is to do that.
Made every single day a death march and a company that survives purely on the sunk cost fallacy on the part of the investors. They've already dropped $MILLIONS into building this thing, and the CEO swears that this next big deal will be what finally pushes the company into the stratosphere. As miserable as it was to work there, I'm honestly impressed they're able to keep that zombie moving.
The leadership at my publicly traded company is a revolving door of short term hacks for promos followed by a quick pivot to the competition or another company. It often feels like the C-Suite at most companies thinks at most, 2 years out.
I just can't see many leaders thinking 5 years out.
I've seen so many developers over the years who view having to rewrite that module because it can't scale well enough anymore as a failure, but I call it a success. Software is meant to be tumultuous, things SHOULD be getting replaced.
In another post in this thread I mentioned that software systems are like children in that they have phases. Assuming that your company and software team is dedicated to fixing those issues (slowly, beside new features), then I think you're probably in a healthy spot despite the pain.
An observation is that business interests must ALWAYS trump technical interests when those business interests are critical. No business = no technical. But at some point when the company becomes stable that balance has to start shifting back towards the technical or the company itself becomes ridiculously inefficient.
The truth is that the world is a complex place and you don't always know whether you can keep your existing customers by improving what you already have or get new customers to decrease the risk of declining income. It is hard to measure what is acceptable tech debt and what is worth addressing.
In one sense, the proof is in the income. If you can help customers with an 80% OK product then you just need to live with the dodgy 20% bit.
I agree with the danger of big customers though! If you ever think something is a way to avoid hard decisions, pivots and annoying some of your customers, then it is too good to be true ;-)
The amount of churn we had was always the smaller clients who were still paying good money but felt neglected and would be open to changing vendors.
We went out of our way to mind the big clients and thought if we lost these few big clients it would so dramatically adverse the companies bottom line, we couldn't afford to lose them. This resulted in a situation where they were able to hold our company hostage, make unreasonable demands on time and energy and hamstrung our ability to keep our more loyal clients (who only bothered us on occasion) happy.
It was a good lesson there is a danger when you have a small company servicing a much larger company that you get into a position where your fear of losing them is a detriment to your other, more faithful clients.
The point of the article is not to convince anyone to slow down as much as possible and work on bugs/etc. It's just that at some point (3, 4 years in?) there comes a time that you just have to put more effort into the things I described, otherwise it gets complicated fast.
Obviously there is a tradeoff, my opinion is that this tradeoff wasn't correctly balanced (especially further down).
Big customers certainly sounded good on paper, but they came with a cost (complexity and a very hands-on relationship) that we weren’t really prepared for, and that we didn’t manage to put boundaries on fast enough.
This is an issue I've dealt with more than once in my career. It seems to occur especially when an organization values the sale more than it values the business. It's virtually impossible to successfully apply boundaries after the customer has been trained that there are none. Irrespective of how carefully you do it or how much you try to reassure your customer, it always feels like you are taking value away. I've encountered this recently in an organization that values customer happiness, which it calculates as an absence of customer complaints, more than customer success. The squeaky wheels get all the grease at the expense of just about everything else.
And the software inevitably has become a maze of settings and IF-THEN-ELSE blocks and SWITCH statements, increasing the cognitive load required to make what should be simple changes. When new features are added, the impulse is to code defensively and build inevitable customization into everything up front. And further down the spiral of technical debt and increasing complexity we go...
Like: don't just add weird features that make sense for just one customer, but are eventually delivered to all customers, but instead maintain a "core product" which provides a lot of generic, core functionality, but which by itself isn't actually usable and just serves as basic building blocks to construct the actual end products which are then built and packaged individually for each (big) customer. Each of these "end products" is a development project in itself, has its own team, its own codebase, but they all draw from a core set of base functionality developed and maintained by another dedicated team that is not directly in contact with any of the customers (or at least not permanently).
This - at least in theory - prevents custom functionality for specific customers from polluting the common core product that aims to please a general audience, while still allowing for the necessary freedom to satisfy big customers demanding custom functionality, but also offering big checks in return.
All else being equal, a smaller company will spend a lower absolute amount to work around imperfections, and get lower value from any product overall. So of course they will not be willing to pay as high a price. Hence, for commodity use cases, use commodity software rather than fully/semi-customised.
This article is in that direction. > … the difficulties started piling up fast, and before we knew it we had to deal with a lot.
When it comes to your customers, I believe that you want to understand as much as you can, as early as you can, about what they are actually trying to do and about how they are actually faring with what you have sold them. I'd argue that the success of your initial customers is more important than the features for your next deal, and that feature investments should reflect that.
The problem, as the OP said, is in estimating the difference between the incremental feature requirements of your initial customers and what your strategic feature set will be. There may be no difference, and your strategy is to ship features in order satisfy each new customer as you get it. Perhaps you can design for that. Chances are though that your first customers are not wholly repsentative of your strategic, target market.
So, in the early stage, you have to make choices about allocating dev for tactical "now" or for strategic "later". In the early stage, if there is not enough "now", there is no "later". The question is when to begin budget for later. I'll argue that you are attempting to build a business, not a suspense movie, and that you budget a fixed amount for "later" from day one. Plan to succeed.
I agree with the recommendation of a strong Professional Services function as an buffer for dev, but will note that culturally you have to work hard to not silo. Your Support and ProServe teams become a primary channel of learning for dev. As a business, though, you may have to give away ProServ hours.
All in all my biggest takeaway here is that you attempt to "hire" customers strategically. As you journey together, will this customer want to go where you will want to go? What time is it when your "Big Customer" sits on your roadmap? ...
"Hire" customers strategically is easier said than done when there's salaries on the line -- accordingly look at how you compensate Sales and budget away from early stage big deals.
Very glad (most of) the cool kids have re-evaluated their stance on microservices.
I suspect good arguments could be made for these things, but they need to acknowledge and justify their cost.
It's tough to suggest detailed solutions to a problem which is in general very vague. Btw I hope the article shows my appreciation for what the company achieved.
One small insight from Uzi (CEO + founder) when I discussed this with him (after publishing article): Base should've focused on much less features but with greater detail. It somehow confirms my guess that we didn't work on existing features as much as we should, but we "spread too thin".
What is it specifically about Feature & Scope creep that you would like to say?