Hacker News Reader

Comment by solatic | Hacker News Reader

That's what works for products, not software systems. Gradual growth inevitably results in loads of technical debt that is not paid off as Product adds more feature requests to deliver larger and larger sales contracts. Eventually you want to rewrite to deal with all the technical debt, but nobody has enough confidence to say what is in the codebase that's important to Product and what isn't, so everybody is afraid and frozen.

Scale is separately a Product and Engineering question. You are correct that you cannot scale a Product to delight many users without it first delighting a small group of users. But there are plenty of scaled Engineering systems that were designed from the beginning to reach massive scale. WhatsApp is probably the canonical example of something that was a rather simple Product with very highly scaled Engineering and it's how they were able to grow so much with such a small team.

View on HN

> Gradual growth inevitably results in loads of technical debt.

Why is this stated as though it's some de facto software law? The argument is not whether it's possible to waterfall a massive software system. It clearly is possible, but the failure ratios have historically been sufficiently uncomfortable to give rise to entirely different (and evidently more successful) project development philosophies, especially when promoters were more sensitive to the massive sums involved (which in my opinion also helps explains why so many wasteful government examples). The lean startup did not appear in a vacuum. Do things that don't scale did not become a motto in these parts without reason. In case some are still confused about the historical purpose of these benign sounding advices, no, they weren't originally addressed at entrepreneurs aiming to run "lifestyle" businesses.

It is a law. The law of entropy.

Try as you might, you cannot fight entropy eternally, as mistakes in this fight will accumulate and overpower you. It's the natural process of aging we see in every lifeform.

The way life continues on despite this law is through reproduction. If you bud off independent organisms, an ecosystem can gain "eternal" life.

The cost is that you must devote much of your energy to effective reproduction.

In software, this means embracing rewrites. The people who push against rewrites and claim they're not necessary are just as delusional as those who think they can live forever.

You don't understand very much about entropy. This reasoning is very, very, very sloppy.

Now I remember why I stopped commenting here.

low-effort comment with ad hominem and zero rationale. fairly toxic.

its not the law but a cost

software is unique field where project can be a problem that no matter how much money you throw, there is something that we can "improve" or make it better

that's why we start something small, a scope if you want call it that way

of course start something small or dare I call it simpler would result in more technical debt because that things its not designed with scale in mind because back to the first point

I think the logic is that good code is code which is maintainable and modifyable; bad code is difficult to change safely. Over time, all code is changed until it is bad code and cannot be changed more. So overtime most code is bad code which is scary to touch.

The Peter Principle of code.

Software is a component of a product, if not the product itself. Treating software like a product, besides being the underlying truth, also means it makes sense to manage it like one.

Technical debt isn’t usually the problem people think it is. When it does become a problem, it’s best to think of it in product-like terms. Does it make the product less useful for its intended purpose? Does it make maintenance or repair inconvenient or costly? Or does it make it more difficult or even impossible to add competitive features or improvements? Taking a product evaluation approach to the question can help you figure out what the right response is. Sometimes it’s no response at all.

Took me way too long to learn this. It still makes me sad to leave projects “imperfect” and not fiddle in my free time sometimes

The discussion is not about the product where you can just remove the stuff. The thread was testing in small setting and then moving to oddball setting. If it is required to cover oddball settings, it makes sense to know and plan for oddball setting.

Designing or intending a system to be used at massive scale is not the same as building and deploying it so that it only initially runs at that massive scale.

That's just a recipe for disaster, "We don't even know if we can handle 100 users, let's now force 1 million people to use the system simultaneously." Even WhatsApp couldn't handle hundreds of millions of users on the day it was first released, nor did it attempt to. You build out slowly and make sure things work, at least if you're competent and sane.

Sure, but if you did a good job, the gradual deployment can go relatively quickly and smoothly, which is how $FAANG roll out new features and products to very large audiences. The actual rollout is usually a bit of an implementation detail of what first needed to be architected to handle that larger scale.

You get certain big pieces correct maybe but you’d be surprised how many mistakes get made. For example, I had designed the billing system for a large distributed product that the engineer ended up implementing not as described in the spec which fell down fairly quickly with even a modicum of growth.

Well, Google got good at large scale rollouts, because they are doing large scale rollouts all the time. _And_ most of the time, the system they are rolling out is a small iteration from the last system they rolled out: the new GMail servers look almost exactly like the last GMail servers, but they have on extra feature flag you can turn on (and which is disabled by default) or have one bug fixed.

That's a very different challenge from rolling out a brand new system once.

The issue with FAANG is that they already have the infrastructure to make these large scale deployments. So any new system - by necessity - needs to conform to that large scale architecture.

The other nice thing about FAANG is that almost nothing they do is actually necessary. If Facebook rolls out a new feature and breaks something for a few hours, it doesn't actually matter. It's harder to move fast and break things if you're, say, a bank, and every minute of downtime is a minute where your customers can't access their money. Enough minutes go by and you may have a very, very expensive crisis on your hands.

Yeah, people get real upset about even 1 messed up money transaction.

Replying to myself in sibling: except maybe people paying for ads, which is more of a faith based action; it's well known a lot of ad traffic is fraudulent, but not which traffic. So if you pay for ads, who can tell what happened.

FAANG tests first on test beds, and on subsets of their user base.

also, see what happened last week when Cloudflare pushed out a bad configuration without trying it on a subset

No but whatsapp was built by 2 guys that had previously worked at Yahoo, and they picked a very strong tech for the backend: erlang.

So while they probably didn't bother scaling the service to millions in the first version, they 1) knew what it would take, 2) chose already from the ground up a good technology to have a smoother transition to your "X millions users". The step "X millions to XYZ millions and then billions" required other things too.

At least they didn't have to write a php-to-C++ compiler for Php like Facebook had, given the initial design choice of Mark Zuckeberg, which shows exactly what it means to begin something already with the right tool and ideas in mind.

But this takes skills.

Did they succeed because of Erlang or in spite of Erlang? We can't draw any reliable conclusions from a single data point. Maybe a different platform would have worked even better?

Yeah - the technology used is a seperate concern to their abilities as users (developers) of that technology and the effectiveness at handling the scale.

I, for example, have always said that I am more than capable of writing code in C that is several orders of magnitude SLOWER than what I could write in.. say Python.

My skillset would never be used as an example of the value of C for whatever

Erlang is uniquely suited to chat systems out of the box in a way that most other ecosystems aren't. Lightweight green threads via the BEAM vm, process scheduler so concurrent out of the box, immutable data structures, message passing as communication between processes.

There's nothing unique about Erlang. I have nothing against it but other companies have built messaging systems using other platforms that work as well or better than WhatsApp.

Examples please.

> No but whatsapp was built by 2 guys that had previously worked at Yahoo, and they picked a very strong tech for the backend: erlang.

https://news.ycombinator.com/item?id=44911553

Started as PHP, not as Erlang.

> 1) knew what it would take, 2) chose already from the ground up a good technology to have a smoother transition to your "X millions users".

No, as above, that was a pivot. They did not start from the ground up with Erlang or ejabberd, they adopted that later.

Thanks, somehow I remembered wrong.

They took already existing protocol (XMPP) and already existing implementation in Erlang (ejabberd)—there weren't many alternatives at the time really.

> Gradual growth inevitably results in loads of technical debt that is not paid off as Product adds more feature requests to deliver larger and larger sales contracts.

This isn't technical debt, necessarily. Technical debt is a specific thing. You probably mean "an underlying design that doesn't perfectly map to what ended up being the requirements". But then the world moves on (what if a regulation is added that ruins your perfect structure anyway?) and you can't just wish for perfect requirements. Or not in software that interacts directly with the real world, anyway.

You have to design for scale AND deploy gradually

Yes, absolutely. Knowing that it will need to get big eventually is important, but not at all the same as deploying at scale initially.

Yes, it can be very difficult to add “scale” after the fact, once you already have a lot of data persisted in a certain way.

Gradual growth =/= many tacked on features. Many tacked on features =/= technical debt. Technical debt =/= "everybody is afraid and frozen." Those are merely often correlated, but not required.

Whatsapp is a terrible example because it's barely a product; Whatsapp is mostly a free offering of goodwill riding on the back of actual products like Facebook Ads. A great example would be a product like Salesforce, SAP, or Microsoft Dynamics. Those products are forced to grow and change and adapt and scale, to massive numbers doing tons of work, all while being actual products and being software systems. I think such products act as stark rebukes of what you've described.

we get paid to add to it, we don’t get paid to take away

Now there is your problem. It is only true in the context of grave incompetence, though. I have worked on tickets with 'remove' in the title.

There's nothing wrong with technical debt per se. As with all debt, the problem is incurring it without a plan or means to pay it off. Debt based financing is the engine of modern capitalism.

Gradual growth to large scale implies an ongoing refactoring cost--that's the price of paying off the technical debt that got you started and built initial success in small scale rollouts. As long as you keep "servicing" your debt (which can include throwing away an earlier chunk and building a more scalable replacement with the lessons learned), you're doing fine.

The magic words here to management/product owners is "we built it that way the first time because it got us running quickly and taught us what we need to know to build the scalable version. If we'd tried to go for the scalable version first, we wouldn't have known foo, bar and baz, and we'd have failed and wouldn't have learned anything."

Reply on news.ycombinator.com