The Prime Video microservices to monolith story

Please share it. And don't rely on articles that are more than a decade and 3 or 4 versions of PHP old.

As far as I am aware, and I can tell, it wasn't because of the stacks / architecture, but lack of product market fit. The fact that the response to lack of PMF was to double down on product features and tech (instead of a business pivot), in my opinion, was what made the investor money go poof and the startup die down. I've seen this happen' in too many places (as an employee or consultant) to be a coincidence.

> And, there is a good reason people shy away from PHP, and it has nothing to do with trying to be "flashy." There should be a name for this kind of fallacy.

Different stacks for different use cases. If I can't get money with a shitty PHP product offering, it's probably best to figure out why instead of attributing it to my tech stack.

The only people pointing to PHP as the right choice that kept them from ruin are those that used it successfully. Therefore, they were building something that fit well with PHP's benefits and didn't expose too many of it's weaknesses. But no one knows for sure exactly what about that project made it a good candidate for a PHP monolith, because we don't really have any science around software engineering, just hearsay and fashion.

Having rescued several 10+ person year PHP projects from ruin, none of them were on the edge because of the language itself. However, the people building those systems made an incredible mess in a short time. I think each language attracts a certain type of developer and comes with it's own community baggage. The PHP community has some highly questionable best practices. But this is true of all programming language specific communities. They tend to be insular and have a lot of "winning the last war" advice for working around the shortcomings of the language's previous major version that have been fixed in the current major version.

(1) https://en.wikipedia.org/wiki/Survivorship_bias

I agree, I also strongly dislike Angular and MySQL in addition to PHP. But I also think a much better indicator for success is discipline. Angular, PHP and MySQL, to their credit, aren’t crazy Turing tarpits, you can write good, successful software with them.

I’ve never seen a dysfunctional project where any of the inherent flaws of those technologies even registered on the same scale as the sheer malpractice of everyone working on the product. And this is also the problem with trendy and flashy stacks, developers not learning the stacks properly and using them as intended. Probably due to commercial pressure to push prototypes to production instead of rebuilding properly after learning the stack. I think that should be the takeaway from the original comment, that CTOs need to either budget for building it twice, or stick to a stack everyone knows inside and out.

I don't think one stack is necessarily better than the other, it's all about how well do the developers know the tech stack in question.

> I doubt there is many devs that want to work on Angular 2 these days.

From my experience, a product that makes money and is able to pay people at market rates, will find said people. Not everyone is an EMACS wielding, Linux wizard that refuses to touch tech out of idealistic concerns.

The problem, in my experience, is more often lack of PMF than tech stack (I've seen customers give insane leeway to actually valuable products).

So if I stay with a boring stack, I'd lose competitive advantage and face the potential of being "un-hirable" in the not-so-distant future.

The people who job hop every 18 months and have flipped from ML to web3 to LLM's in the past 4 years are not the people you want to hire. If someone has 5-6 years experience writing Java code, they'll be fine if your tech stack is C#, and vice versa. There might be a bit of a learning curve pulling people from python or rails onto typed languages (or vice versa), but in my experience that's something that can be overcome easily enough. If you're willing to give someone a month to ramp up with the expectation they'll stay longer than 2 years, that seems to be a nice tradeoff to make.

We used to build real-scale systems with a stack of this kind when we had fewer features and less CPU power.

When you are designing replacements though you need to have an idea of what your scalability runway is before the next re-design will be possible. Sometimes that is just 2x, often times it's 10x, occasionally it's 100x but it's all situational.

Even for "peak" Amazon's concern seemed limited to about ~5x daily TPS maximums unless one had extraordinary evidence.

The counter-balance to limiting resource balooning to 5x scale is introducing Erlang-B modelling. Depending on how many 9s you require, you may need way more service availability than expected.

The 100x calculations are probably doubly wrong (both too large and too small), providing negative value false confidence.

My current employer is going all-in on serverless because it solved a problem they had with performance. The problem wasn't solved by serverless, but by moving away from an old fashioned and unscalable solution. The real problem or bottleneck - a centralized SAP database - has not been solved yet. They would have achieved the same results if they rebuilt their API in a generic Java monolith.

Here's my prediction: when the crack team of consultants that powered through building a serverless architecture leaves (because they will, one because they're consultants and two because they get bored when the problem has been solved), they won't be able to find competent developers to take over and the whole thing will be rebuilt again in something they CAN find developers for. I mean it's just NodeJS, but the architecture is distributed and much harder to manage than in a simple monolithic app.

However, what happens is most of the time the product or company will never even reach that scale so that effort will be wasted. If you're not able to close the loop and actually support a running system at scale, with a real workload (not simulated or ab stats) it actually does the opposite: you don't learn what really works and what doesn't. It's easy to convince yourself you have built a system that can handle such scale based off back of the envelope calculations, synthetic testing, or cargo-culting.

Most of what actually happens with scale in the real world has to do with old & boring things like appropriate choice of data structures, cache design, load balancing, locality, etc. and has nothing to do with distributed computing, serverless, etc. which are just tools that have specific characteristics that might make them good choices or not.

Just look how over-represented RoR is/was as bootstrap tech in known, successful companies.

Microservices is not yes/no - it's a slider. You may find sweet spot at ie. 50/50 split like ie. github does, have more or less microservices while keeping core in monolith or services under single versioned monorepo.

Microservices are good for satellite services like system integration, pre/post-processing, gateways etc.

As a rule of thumb whatever can fit single (tech lead + team)'s "head" - can be monolithic (single monolith or set of services under single versioned monorepo managed by that team). Their job is to provide stable apis/uis - otherwise it can be seen as black box by other teams.

This is natural way things evolve (teams are created around naturally bounded concepts) and the suggestion is - don't break it by creating mismatches, keep it in harmony. If something creates measurable problems, slowly form team around it with new tech lead from existing team and let it grow on it's own - it'll extract itself to separate subsystem by itself. It doesn't have to be done overnight.

It's astounding how many people use "scale" as chupacabra to scare everybody in the meeting room without clearly defining what they mean by "scale". To make decision around changing architecture to scale you need to precisely define what it means, have metrics, have benchmarks showing current limits, proof it's a problem now or near future and focus just on those actual issues, if they even exist.

I am disappointed that the top comment and all of the discussion is once again not related to the article at hand in the slightest and just focused on the easy "complex is bad, guys!" dunk.

To combat this, you need to justify the complaint.

In the case of lambda, I think being tied into an ecosystem of things that you can't run locally is a net loss, but Amazon has done a terrific job at making a bunch of people not super comfortable with compute, see just running a share-nothing process as the way to bridge the gap, like PHP never existed.

Same thing happens with IT orgs that end up owning cloud. You get azure, regardless of whether you want such a thing.

It seems that most of the takes against this article are coming from people that have opinions about AWS based on nothing.

Sure, AWS is selling Lambda/Serverless services. Sure, they are making articles/videos on microservices architecture. But they are also selling EC2 instances. They are also making articles/videos on monolithic architecture.

I have never come across any content from AWS telling that this service/architecture/mindset is the best and every company should use/implement it.

Anyone who went through the Well-Architected Framework would be completely aware of that.

But I guess it's easier to mock some folks over the Internet than taking time to grab some knowledge and reflect.

Once you're designing something beyond a small app, you'd have to pass PE review anyway.

Rational take, but I see the debate similar to Roman vs Arabic numerals.

Keeping a tally? Roman. Need to use operators? Arabic. Sometimes you can keep a tally in Arabic (not ideal), and sometimes you can do basic operations on Roman numerals (not ideal).

However, when you want to start using variables, only one tool enables this easily.

I can't architect the types of redundant and properly separated interoperable systems with a monolith that microservices otherwise enable.

So the desire to move forward isn't the need to find a magic bullet, but the next evolution of an existing ability that unlocks new capabilities...

(I don't think calculus would have been discovered using Roman numerals)

“You get double the thrill if you think you are engaging in logical reasoning, all the while arguing anecdotal experiences, beliefs and biases.”

Not only that, but if you absolutely must (for example different scaling requirements), forking out a service from a monolith and deploying it as microservice is relatively straightforward. Provided source code interdependencies have been maintained cleanly before, which can be done with support from tools.

> To know this enough experience (both good and bad) is needed with both microservices and monoliths.

Over the years I have worked with a huge number of people who are looking for Silver Bullets and Best Practices, generally for one (or both) of these reasons:

- To be blameless if the architecture doesn't work out (building with microservices is a Best Practice, it's not my fault it got very expensive)

- To cut down on the number of deep-thinking decisions they have to make

And to be fair on the second point, I don't necessarily disagree with the premise; if these kinds of decisions allow you to get going quicker, it's not necessarily the wrong choice at the moment. What everyone needs to keep in mind, though, is that if you don't do deep architectural thinking and planning at the start of a project (or even if you do sometimes...), you'll have to revisit it later. Often the same forces and pressures that led to taking shortcuts at the early architectural stages of a project will continue to be present down the road, and it'll be more work to fix it later than to do it right the first time.

It's a tough balance. If there's a high risk that the project won't be alive in 6 months it probably doesn't matter. If it's a prototype that's very likely to get into and stay in production, then it really does matter.

The problem comes when you try to start a new project and begin to realize that except in simple cases, it's exceptionally hard to know ahead of time how you would best be served to split an application up into services. Once there's a network boundary between two pieces of code, there's now an API that needs to be managed carefully: deployments of the two pieces of code may not be in lockstep, rolling one back may be necessary, etc. If you got the split wrong, the refactoring to fix that could be extremely hard.

By comparison, splitting logical services out of a monolith isn't always a cakewalk, but it's usually more straightforward, especially since as you mentioned, it's totally possible to just share the same codebase between services. It could even be the same binary invoked differently.

My feeling is that it's easier to split a monolith than to join microservices that were developed separately together; they might have different codebases and possibly even be written in different programming languages. Therefore, it's probably better to start with a monolith even if you're pretty damn sure you will need some services split off later to scale.

Serverless on the other hand I am more skeptical of. I like certain products that provide generic-ish abstractions as a service: Cloud Run for example seems like a good idea, at least on paper, and Amazon Aurora seems relatively reasonable, too. But I think that they are often too expensive and do not provide enough value to justify it, especially since a lot of serverless tools don't provide terribly generic APIs or services, leading to substantial lock-in.

Both microservices and serverless APIs were sold as near-panaceas to the woes of traditional monoliths, but it's a lot more complicated than that.

edit: When I typed this response, the message I was replying to was different/much shorter.

Someone needs to make a programming language that encourages you to write code that can be easily split and recombined. Something that can do remoting well.

What I am saying is the highest quality tools are still the best and will remain that way. As other things come and go they absorb the best ideas from the new stuff without losing what makes them good.

Sometimes the new stuff survives (Node/TS/Go/Rust) sometimes it doesn't (Clojure, Scala, arguably Ruby), some we are waiting to find out.

But betting on JVM and .NET has been winning strategy from a technical perspective for a long time now.

Also worth mentioning no, PHP probably doesn't work here because it's not "general purpose" enough IMO. JVM can handle almost any workload shape, the main one it doesn't do well in is memory constrained environments (though it's possible if you really want to). PHP, Node/TS, Python, Ruby all suffer from being single threaded so can't take advantage of properly large machines without a ton of extra complexity and you pay a high memory overhead for most of those strategies.

Go is OK, I hate writing it but it can do all the things so if that floats your boat go for it. Rust is also perfectly fine, I just find it is actually pretty slow to write because changing requirements often mean changing structure which can be harder to manage in Rust vs Java/Kotlin/C#.

Anyways, wasn't meant to be a thing about languages. Just pick a good runtime or AoT lang of choice, write everything in it, stick to conventional frameworks, choose PostgreSQL, run the lot on k8s. That is the core takeaway.

PHP can't run everything though. If we go by the traditional model it's per request and fails at things like websockets. It's also less performant than a JVM solution, so definitely can't serve every use case.

Also I find it way easier to convince people coming from a dynamic language background to try it out vs Java which they find less approachable. Which is ironic as Kotlin is by far the more complicated language vs Java.

The only way to learn Spring is to read the latest manuals of every subproject you think you might need, cover to cover.

Even then you might not figure out what awkward collection of beans you need to override the magic Spring does.

k8s is funny to call "boring tech" because it's notoriously complex. But if your company has set up dozens of k8s clusters and has a bunch of k8s experts on staff, it can be considered "boring tech." It's all relative.

"infinite" (meaning wallet level) scalability.

Integrated deployment tooling usually.

Integrated ingress usually.

Those last 2 really don't matter though as neither problem is that difficult if you know what you are doing. They both would have been a huge selling point in a world where k8s doesn't exist. Alas it does and thus mostly moot.

The first however is a big deal as managing very bursty loads on k8s can actually be a bit of a challenge. This is doubly true when you are using some sort of queue and don't have good queue scaling metrics for whatever reason. Serverless lets you cop out of managing that by using infinite concurrency.

Ok so now cons of serverless:

Turns all your code into spaghetti, this is probably the biggest one.

Cold boot performance is highly varible and generally worsens with added function complexity but this depends on tech choice.

Infinite concurrency is a bit of a curse. Want to use that MySQL DB that contains all your business critical data? Better use some MySQL Serverless proxy nonsense because unbounded connections isn't fun.

Lock-in is awful, migration story is awful.

Amount of Infrastructure as Code overhead, ala Terraform, CloudFormation, whatever else floats your boat to support serverless is insane. The sheer number of objects that need to defined, kept in sync etc is just bonkers. Not to mention if you have multiple services setup like this it's all boilerplate you either need to abstract or carry around.

Hides serious programming deficiencies like memory leaks because context is thrown away after servicing request.

Unpredictable runtime makes observability more complicated, probably forcing you into poor decisions as a result, i.e prom push gateway.

Runtime/execution time limits mean people trying to do all-serverless end up doing horrible things to accomplish long-running tasks.

All of the above leads to increasing pressure to make poor architectural decisions to work around limitations of the runtime.

I could probably go on but that is a reasonable set for you.

EDIT: Actually I forgot the worst con. The cost. The $/CPU/RAM cost of Serverless is beyond bonkers. When I was pricing up Cloud Functions vs Spot instances on GKE I think I came to the conclusion Cloud Functions are about 300x as expensive. If your code can run on serverless it can probably run on Spot instances which you can just add to your GKE deployment using another node pool and only schedule things that can handle nodes going away within 60s there and instantly reap 300x cost savings. This is because Serverless is sold at a massive premium to CPU list prices and Spot is sold at like ~30-40% of list prices depending on machine type (for those curious when I last checked T2D was the best for price/performance for highly single thread performance dependent workloads, ala Node.js)

Boring tech is usually more stable and more mapped out. Everyone knows where its good and where its not. Other people have already had whatever problems you have.

I'll always bet on the boring tech over the hyped exciting tech.

I've been learning k8s fundamentals recently after using borg for a long time and then awhile kind of just poking at k8s trying to use it without learning it. Honestly I think it seems like about as simple as a solution to a broad set of real common problems could plausibly be.

IMO if you are facing those problems it has to be poor leadership, there is no excuse for why you shouldn't be able to make effective decisions if you know what you are doing.

Also generally Python is just used for terminal processing in the Big Data world, all the pieces in between are JVM anyway.

We are looking at taking our .NET6 monolith and "inverting" it so the entry point is our logical HTTP resources instead of the various DI monstrosities that emerge from main.

The reasoning for us is that we don't want ownership over all of the horrible things we could get wrong. If a <TLS1.2 connection some how gets established and something naughty happens because we forgot to update some startup.cs boilerplate, it's currently our pain today. Like legally. If we are just eating pre-approved/filtered/etc HTTP requests from inside our hermetically-sealed function environment, we get to cast a lot more blame than we otherwise would be able to.

I really don't get why you'd go to serverless and then shit it up with a lot of extra complexity. The whole point in my view is to inherit the compliance packages of your hyperscalar by way of directly using their native primitives in as standard a way as possible. This means you get to 'sail' through audits and engage up-market clients that are typically out of reach for everyone else.

However if you have a usecase that is absolutely a perfect fit for serverless i.e incredibly bursty, unpredicable, very short runtime. To the point it would be to the detriment of other services hosted on the same infrastructure i.e would need say an isolated cluster anyway then yeah, serverless could be a cost worth paying.

Kubernetes is much more open standard. It's much easier to move to other cloud or self-host.

It's more about insurance and keeping other options.

Managed Kubernetes, managed databases are a thing and you should use them if you can. Just don't depend on cloud-specific stuff too much.

I have used (at large scale) GAE, Google Cloud Functions and Lambda, before that I also used all the previous "serverless-like" things including Beanstalk (anyone else even remember that?), Heroku and bunch of other PaaS things for various stuff.

It's all come and gone. To me serverless is just the latest in a long line of things that I get paid to replace with things that actually solve the problem and doesn't turn into a ball of spaghetti that no one is able to modify anymore.

For me it's actually the decision of a CTO of one of top 10 companies in the world in terms of software size/value/etc. to push it down to very smart people - that's bad leadership.

Why? Because I guess this thing didn't go as planned - a self fulfilling prophecy that shows how awesome serverless is. It backfired in a period where people are questioning whether all these megahypes are not actually just some marketing. Yes, he is responsible for that, for pushing it down to his people that "everything new must be started with serverless" first. When actually in some situations, as the article showed, a monolith or a (big) microservice would do the job better.

Why the hell hire super smart people when you have to tell them how to do things?

The OP's comment "Also generally the conception that building things with containers + k8s actually takes more time than serverless is a horseshit take IMO. It doesn't and if it's taking you longer you are probably doing something wrong."

I couldn't disagree with more from a hands on practical perspective. That will 100% take anyone longer to spin up for an MVP (for more cost) than spinning up a FreeBSD server and I'm unaware of anything that needs more than that for an MVP.

This isn't even mentioning the fact that every second your data sits in an AWS/CGP/Azure datacenter - you are not only telling them what kind of data you have (don't pretend like the hosts don't know and can't tell) but you're increasing your switching/egress costs

Or fantastic engineering leadership.

OP has constrained the technology choices of the org into a very capable set of primitives. The entire org can move faster by sitting atop the same rails.

K8s + tried and true tech lets the org spin up additional compute anywhere and easily migrate off a particular vendor. Lift and shift is work, but not nearly as bad when the rails are simple.

He then fails to apply the same rule to k8. Almost nobody needs that either and treating it as the default not a good strategy.

Maybe charitably assume it’s based on years of experience doing this exact thing?

It’s certainly my experience.

Competitive advantage comes from understanding user needs, great UX, and a smooth running sales machine. No-one cares that you built with technology X, whatever that is. There is no silver bullet.

How about I clarify that my stance mostly relates to Functions as a Service so it's clear that while I wouldn't choose Cloud Run over GKE that is hugely different to FaaS.

I was working on a competitor to k8s at the time though so my knowledge of the subject matter definitely influenced my opinion of it.

I thought it was unstable though, largely due to dependency on etcd which was also unstable at the time. Thankfully that got sorted out though.

I’ve also been at this 40 years and what you ended with is worded unnecessarily crass, in that folks of course learn perspective over time.

That isn't making a choice and adjusting based on the data, that is just shoddy engineering.

It should have been obvious to anyone at a cursory glance that not only did serverless not offer anything to this application it would be actively detrimental. Simple napkin math of data rate of a 1080p video stream should have made that immediately clear.

Just because you can use something to get a task done doesn't make it a good choice. I can still write pretty decent Perl. It gets the job done and for some protocols that haven't changed in a long time like LDAP the existing CPAN libraries are still some of the best. But that isn't the right choice is it?

I don't think it's defensible to argue that we should build thing sub-standardly just to give people more choice. That is like arguing that real engineers should have more freedom in choice in how they design bridges. No one would agree with that.

We get away with it because our bridges are much cheaper to replace and noone dies if they break or have to be rewritten because they rotted too much to run anymore.

Right now as it stands arguments for serverless aren't technical in nature. They are a best an appeal to "the traditional way is slow" or some notion of "infinite scalability", the first of which isn't true at all and the latter doesn't matter for anyone that is arguing in favor of serverless.

Also even if it was "faster" (I don't agree but lets postulate) that probably wouldn't make it a good decision either unless it was proportionally faster vs it's costs (not just dollar costs, architecture, complexity of managed objects, etc). Which I highly doubt would be the case.

As you can see from a technical POV it's just very very hard to rationalise a position for serverless > Boring Tech. For me it would take too much mental gymnastics.

I was perhaps a little strong but my points are solid.

If it was good it would have proven advantages by now. K8s adoption curve vs serverless adoption curve tells the real story of quality IMO, especially given how much bigger of a move it was to k8s from what preceeded it.

Both are complex internally, k8s does a better job of preventing that complexity from leaking IMO.

You need to know roughly the same amount of things to use both.

On k8s to deploy something simple you will need:

1. A deployment to actually run your app.

2. A service describe how to talk to your app.

3. An ingress to route requests to the service.

On Lambda w/API Gateway:

1. Lambda Function

2. IAM Role

3. API Gateway objects (Deployment, RestApi, Stage)

More if you expand out the policies and API gateway objects. These become -really- burdensome if you are using IaC like Pulumi or Terraform.

In practice using k8s these days is essentially managed also, ale GKE/EKS/AKS so the real differences are down to:

1. API, the FaaS etc ones are proprietary, k8s is standardized.

2. Runtime environment and execution limits, varies by FaaS, k8s has none.

3. Access to durable storage, FaaS only object/HTTP based storage, you can attach EBS-like to your k8s containers.

4. Architecture incentives. FaaS highly incentivises microservices, k8s isn't really opinionated. FaaS is more like cgi-bin on steroids, k8s is daemons on a box but with fancy management.

5. Scaling. FaaS scale "infinitely", k8s scales depending on settings of cluster autoscaler.

So generally you get as good or better from k8s, the API complexity is about the same except now it's standard no lock-in, no enforced runtime limits, you have durable containers so you can attach a debugger, etc.

Hopefully this shows a bit why I think k8s is at least as simple as serverless if not more simple.

And maybe this again proves the initial comment's point.

> I do agree that there is a misconception with containers taking longer than server less. I don't think either takes longer considering the current tooling available.

And (the misconception) that it costs more.

> AWS Lambda and MongoDB Atlas

If you take a managed database with a free tier and compare it there are similarly managed databases (SQL) based with a free tier. Not really fair to say PG will cost more.

This also comes down to the world is not just AWS. k8s is cheap in a lot of places and definitely not hundreds a month.

IMO managed container runtimes are PaaS, are both FaaS and PaaS serverless? Who knows, we don't seem to have hard definitions but in my mind PaaS predates serverless becoming a fashionable word so I associate with FaaS and FaaS-like things only.

This has nothing to do with "serverless". Heroku did the same thing well before serverless was available.

Keeping a simple app running on heroku over the past 15 years has required about 3 hours admin total since I set it up (moving from an old stack to a newer one).

We have a legacy Go microservices codebase that’s over a decade old and because the Go maintainers have paid so much attention to avoiding creating breaking changes and a guarantee to never create a “Go 2”, the codebase has aged much better than stuff written at the same time in Python (some of which is still in Python 2 because it is too risky to upgrade, and that functionality will be killed instead because we don’t know how to safely update it). Most of the time we don’t need to make any changes when bumping Go versions, just fixing up things our linter warns is deprecated.

Twitter is where you get the JavaScript ‘influencers’ who appeal to a core demographic of engineers with FOMO who signal their ‘in status’ by waxing lyrical about the latest framework

And add to that the zero cost of capital of 2010s, it lead to engineers being given almost free reign which is generally a bad idea. “we are a technology company” would attract VC money, as if tech itself was a product of every darn startup.

The software industry is filled to the brim with midwits who think their idea is the only right idea even though its nothing special and they’re ignoring actual problems

As op said - when’s the last time you heard someone talk about writing code that’s easy to debug? Never. Yet 90%+ of companies will make you pass arbitrary algorithms tests

The field is dominated by A types who confuse their neuroticism and ego for talent

I don't mean this in a bad way, but people not part of <group> need to dial it down a bit. Too many get offended by things that people in <group> don't see as a problem.

This is the main problem.

Excluding VC, businesses exist to make profit. The purpose of IT and engineering is to enable that business or make it operate more efficiently - engineering and technology should never be a goal in and of itself.

Given this situation, the most efficient technology (that enables faster delivery of business software) would win. "Shiny" stuff would never win in terms of mindshare unless it's actually good and enables faster delivery than the status quo.

Now add VCs to the mix. Suddenly you have an infinite supply of free money that distorts the usual market dynamics, allowing businesses to keep operating with no regard to profitability. The downwards pressure of the market pushing for businesses to operate more profitably is no longer there. As an engineer, you no longer have the pressure of building technology for the purposes of making the business more efficient - you lost that valuable feedback loop.

Now run this for 10 years. Entire careers will be built on this - why even try to build a profitable business when you can be a "serial startup CEO" and merely use the right amount of shiny to solicit endless VC funding to continue the adventure? Imagine a well-meaning, budding engineer joining such a startup and learning such a corrupted idea of "engineering" with none of the usual objectives, pressures and feedback loops of engineering.

In the end, you end up with "senior" engineers (with over-inflated titles) whose entire careers are built on technology that wouldn't actually pass muster in an actual business under the normal market pressures. But they don't even know anything else or why that is wrong (because they've started their career during this charade and never had any taste of what real engineering is), so they keep pushing for these (now-inadequate) solutions, deluded that they are doing the right thing.

Things like serverless appeal to businesses due to the much smaller logistical implications it presents to a business. Some businesses enjoy that (at least as promised), It’s all taken care of and the team you need to manage it is likely to be smaller. It’s pay as you go for what would otherwise imply more human beings and contracts on your plate.

I imagine at some point those baked in costs and redundancies, the luxury of having huge scale on demand in a general sense, will be much higher than something thoughtfully designed when a company has understood their needs well enough after operating at scale.

For what it’s worth I prefer bare metal clusters :)

The stack doesn't exist today. It could have if we didn't jump so quickly to new technology and all these different prog languages.

I've just never heard someone pitch new tech with: "easy to debug and change". It's always: look at this contrived example with some new esoteric feature.

The best thing about this phase is you just get so much shit done instead of screwing around with 10 half-baked over-marketed frameworks.

A lot of packages you never care what’s underneath if they are maintained and do the job. But when you have to manually patch something and it’s a pile of crap underneath then it sucks.

I much prefer to write things myself if I can now instead of using an existing package or framework. Having my own simple packages in my monorepo that I can instantly change is far better than new third party packages.

I start to wonder if checking in third party modules is the way to go.