Comment by captainkrtek | Hacker News Reader

Comment by captainkrtek | Hacker News Reader

Parent Full thread

Reflecting on the last decade, with my career spanning big tech and startups, I've seen a common arch:

Small and scrappy startup -> taking on bigger customers for greater profits / ARR -> re-architecting for "enterprise" customers and resiliency / scale -> more idealism in engineering -> profit chasing -> product bloat -> good engineers leave -> replaced by other engineers -> failures expand.

This may be an acceptable lifecycle for individual companies as they each follow the destiny of chasing profits ultimately. Now picture it though for all the companies we've architected on top of (AWS, CloudFlare, GCP, etc.) Even within these larger organizations, they are comprised of multiple little businesses (eg: EC2 is its own business effectively - people wise, money wise)

Having worked at a $big_cloud_provider for 7 yrs, I saw this internally on a service level. What started as a foundational service, grew in scale, complexity, and architected for resiliency, slowly eroded its engineering culture to chase profits. Fundamental services becoming skeletons of their former selves, all while holding up the internet.

There isn't a singular cause here, and I can't say I know what's best, but it's concerning as the internet becomes more centralized into a handful of players.

tldr: how much of one's architecture and resiliency is built on the trust of "well (AWS|GCP|CloudFlare) is too big to fail" or "they must be doing things really well"? The various providers are not all that different from other tech companies on the inside. Politics, pressure, profit seeking.

Well said. I definitely agree (you’re absolutely right!) that the product will get worse through that re-architecting for enterprise transition.

But the small product also would not be able to handle any real amount of growth as it was, because it was a mess of tech debt and security issues and manual one-off processes and fragile spaghetti code that only Jeff knows because he wrote it in a weekend, and now he’s gone.

So by definition, if a service is large enough to serve a zillion people, it is probably big and bloated and complex.

I’m not disagreeing with you, I liked your comment and I’m just rambling. I have worked with several startups and was surprised at how poorly their tech scaled (and how riddled with security issues they were) as we got into it.

Nothing will shine a flashlight on all the stress cracks of a system like large-scale growth on the web.

> So by definition, if a service is large enough to serve a zillion people, it is probably big and bloated and complex.

Totally agree with your take as well.

I think the unfortunate thing is that there can exist a "goldie locks zone" to this, where the service is capable of serving a zillion people AND is well architected. Unfortunately it can't seem to last forever.

I saw this in my career. More product SKUs were developed, new features/services defined by non-technical PMs, MBAs entered the chat, sales became the new focus over availability, and the engineering culture that made this possible eroded day by day.

The years I worked in this "goldie locks zone" I'd attribute to:

- strong technical leadership at the SVP+ level that strongly advocated for security, availability, then features (in that order).

- a strong operational culture. Incidents were exciting internally, post mortems shared at a company wide level, no matter how small.

- recognition for the engineers who chased ambulances and kept things running, beyond their normal job, this inspired others to follow in their footsteps.

Reply on news.ycombinator.com