Hacker News Reader

Comment by jedberg | Hacker News Reader

I am one of the biggest proponents of microservices. I helped build the platform at Netflix, I've literally traveled around the world extolling the virtues of microservices.

But I also advise a lot of startups, and you know what I tell them nearly every time?

Build a monolith.

It's so much easier to start out with one codebase and one database, and that will scale for a while. Especially if you use a key/value store like DynamoDB (although you will lose relational functionality that can be helpful at the start). And did you know that you can deploy a monolith to Lambda and still get all the benefits of Lambda without building services?

And then, when you start growing, that's when you break out an independently scalable part of the system into a microservice (with it's own data store!). The one that may need to scale independently, or that you want to be able to deploy separately.

Microservices takes at least 25% of your engineering time just maintaining the platform. It's not worth it unless you can recoup that 25% in efficiency.

View on HN

"A while" is underselling it. As long as you have people who are half-decent with SQL, "just put Postgres on a big db server" will get you to 50 million row tables before you have to start thinking about even hiring a real DBA.

50M is something that can easily be done with just a dev who understands that sql is more than select, insert and update with the manual, google and chatgpt.

You can get really damn far with a fat postgres box.

The problem with these discussions is that you can always get a lot out of any given architecture, structure, db approach or whatnot. Can. There's just a lot of daylight between "can" and "likely will."

Ultimately, everything has its limitations and tradeoffs. If we respect them, it's generally a smooth ride. Problem is that we rarely do... within a company (startup or otherwise) under real conditions. There's also a dynamic where we build until the point where something stops us. Tech debt, complexity, over-engineering, under-engineering, feature bloat or antagonism between early decisions and current goals.

There's a self-regulating aspect to this. If architecture is spot on, perfect for the task at hand we can move faster to reach the point where it no longer is.

You're right of course.

My point is that postgres is, compared to almost everything else, easy to get from "can" to "likely will" with just somebody with a brain, a manual and google. In absolute terms it of course depends, but the point is relative.

A brain, a manual and google is usually a pretty good way generally to make solid decisions, respect the limits of your chosen stack.

What happens when there is >1 brain involved... or when brainless, manualless decisions eventually get made... or two pivots from now...

I'm not disagreeing with your approach. I agree with it, especially as starting point. I'm cautioning that resilience against complexity isn't about how easy it is to make good decisions when you understand the spec, read the manual and calmly proceed. Complexity and fragility accumulate when one or all of these are absent. How easily you can (and thus inevitably will) make a mess... not how easily you can keep it clean.

IRL situations with regular rdbs, a very common trend seems to be long term drift between schema and spec. The flexibility and approachability of postgres often enables a lot of kludge eventually.

Data stores have this dichotomy between "look how easy" and "is limiting factor" that speaks to difficulties we don't know how to articulate or isolate.

Of course if there are lots of bozos in the startup then even getting to and maintaining 50 million rows is going to be very difficult.

But if you have a small group of folks that are at least as competent as the folks at WhatsApp pre-acquistion, then there really shouldn't be any doubt whatsoever.

> a small group of folks that are at least as competent as the folks at WhatsApp pre-acquistion

That's a success case... beware survivorship bias.

> if there are lots of bozos in the startup

No arguing that quality engineers are fundamental to quality engineering. That said... by this standard, there's no point in having this entire discussion. Every good db/store out there is good. They all work very well if used as they should be, with due respect to tradeoffs. Yet, almost everyone has db problems. Almost every one of these problems occur well within the technical limits of postgres or whatnot.

"It shouldn't be a problem" when it usually is irl is tunnel vision. There is an empirical reality disagreeing with you. Walking into it with "this shouldn't be a problem unless everyone is a moron" is bad strategy. If you can't think of reasons why architecture can and will become a problem, then just assume that you (or some of you, some of the time) are morons, and try to make it moron proof.

> That said... by this standard, there's no point in having this entire discussion.

Yes, I agree just repeating tautologies is unlikely to be meaningful.

The real tragedy is this clever people usually talking about their usecases and those present the requirements. There is no average system out there.

"There is no average system" is such a good insight.

I've been on both the sysadmin, development, and hiring sides, and with data models at scales of 50M+ records, devs who "understand that sql is more than select [...]" are rare in my experience, as they're a cross between db admins and developers.

Administrating (in particular, query planning and production operations) databases with tables sizes with magnitude of 10M and more records is challenging, and requires a skill set that is very different from pure development.

One won't get "really damn far with a fat postgres box", unless they're doing very simple SELECTs, which is not the case with modern web apps.

> Administrating (in particular, query planning and production operations) databases with tables sizes with magnitude of 10M and more records is challenging, and requires a skill set that is very different from pure development.

Is it more or less challenging than the alternatives? Is it less challenging enough to add a new tech to your stack, add the required knowledge to the team, etc?

I mean, knowing "enough-to-perform-CRUD" SQL is table stakes for developing on the back-end, but knowing $CURRENT-FLAVOUR-NOSQL (of which there are multiple products, all with such substantial differences that there is no knowledge transfer between using them) isn't, so there's going to be ramp-up time for every dev, and then every dev that is added to the team.

I'm not disputing your argument, I'm just pointing out that, sometimes, it's easier and faster to upskill your PostgreSQL developer to "scale the DB" than it is to teach them how to properly use, maintain, architect and code for DynamoDB and others.

It's not just $CURRENT-FLAVOUR-NOSQL. It's also doing custom transactions and locking on top and/or thinking in terms of eventual consistency. It's so, so much more complex than just SELECT FOR UPDATE/BEGIN TRANSACTION.

> are rare in my experience, as they're a cross between db admins and developers.

That's fine. You only ( ;) ) need one on the team.

It's not even funny how many software engineers just don't know that, SQL is crazy fast and performant, if you have basic understanding about it - I was once refactoring (or, rather, getting rid of) a microservice that was just a JSON blob storage on top of Postgres, without having any schema for blobs, with 100s of 1000s of them, no indices, and main complaint was - it's slow.

Ignorance of database basics is something I allude to in the article, but it's a larger topic that I might vent about later.

True. "While" for some large value of time. And if you set up the auto-vacuumer correctly from the start, you can go even further!

Unpopular opinion: if you're skimping on ops/DBA resources (as you may need to do in a startup), then MySQL is a better default. By all means use postgres if your use case demands it, but personally I find the ops story for MySQL takes less engineering overhead.

Yes, and most successful companies who started in the last ~20 years started ( and many continue ) with a monolith and a MySQL database.

Only the mega-cap ones started to pursue other options mostly due to their type of business and bucket-loads of "free" VC money with explicit orders to burn it and get "unicorn" status - which involves hiring thousands of developers in record time and the whole thing turns into a zoo. Which is an organizational problem, mostly not a tech one.

Other than the ones we pretend are the whole Universe, there are thousands and thousands of medium to big companies with billions of revenue who started their product with a monolith and a MySQL database and many still do just that.

I agree with this "unpopular opinion". Worked with both MySQL and postgres based mid-scale apps of several thousands of users. Postgres is so deeply lauded here at HN yet requires two more magnitudes of operations work to keep it up and running. Vacuuming sucks hard.

That's not wrong, but postgres allows for cramming a lot of functionality in the db, and it's fast at following storage trend, i.e. with pgvector

That is a very specific use case, and might only be a small subset of your actual data. If you don't have these specific requirements (eg. CRUD apps), you can save yourself a lot of unnecessary headaches by defaulting to MySQL.

My main point is attempting to counter the narrative popular on HN that postgres should be an automatic default. For sure there are many aspects in which postgres is superior, I absolutely do not debate that, especially when it comes to developer experience. But there is much more to it than that when it comes to delivering business value. That's where ops and DBA concerns start to matter, and IMO MySQL is so far ahead in this regard that it outweighs all the other hideous warts of working with it, when you consider the bigger picture of the business as a whole.

Yes! Vanilla MySQL and a good ORM (peewee, anyone?) gets you extremely far.

It’s so cute that you think 50M rows is big. Your phone can handle many times that, and update it tens of thousands of times per second.

The problem isn't storing or inserting 50M rows, it querying 50M rows in non trivial ways. And the difference in performance between doing that 'right' and 'wrong' is orders of magnitude.

Eh, intelligent table design should knock most of that out. If you've got an 8 page query implementing a naïve solution to the knapsack problem (I've seen this in the wild) several mistakes have been made.

doing it on your phone only has one person querying it. Scale that to several thousand and it might appear different.

I've scaled 300M rows in just one of many similarly sized tables to 1M users... in 2007... on a single box with spinning rust in it. Heck, my laptop could handle 100x the production load.

It amazes me that my comment (while admittedly flippant) got voted down.

It really is true that your phone can update a 50M row table about 10K times per second!

That people are incredulous of this is in itself a stunning admission that developers these days don't have the faintest idea what computers can or cannot actually do.

Just run the numbers: 50M rows with a generous 1 KB per row is 50 GB. My iPhone has 1TB of flash storage that has a random access latency of something like 50 microseconds, which equates to 200K IOPS. An ordinary NVMe laptop SSD can now do 2M. Writing even 10K random locations every second is well within mobile device capability, with 50% headroom to "scale". At 1 KB per row, this is just 10 MB/s, which is hilariously low compared to the device peak throughput of easily a few GB/s.

I believe you are downvoted due to the "how cute" part, not the scale part. You could have delivered useful info without a put down

It's not that good usually, e.g. PostgreSQL writes data in pages (8 KB by default), and changing 10K random rows in the 50M rows table can be quite close to the worst case of 1 changed page per changed row, so 8x of your estimate. Also need to multiply x2 to account for WAL writes. Also indexes. It's not hard to hit a throughput limit, especially with HDDs or networked storage. Although local SSDs are crazy fast indeed.

Agreed: 80MB/s for the random 8K page updates. However, transaction logs in modern databases are committed to disk in batches, and each log entry is smaller than a page size. So a nice round number would be 100 MB/s for both.[1]

For comparison, that's about 1 gigabit per second in the era of 200 Gbps networking becoming common. That's a small fraction of SSD write throughput of any modern device, mobile or not. Nobody in their right mind would use HDD storage if scaling was in any way a concern.

[1] Indexes add some overhead to this, obviously, but tend to be smaller than the underlying tables.

> spinning rust

That's your reason right there, you used rust! It's both performant and secure!

> As long as you have people who are half-decent with SQL

> And then, when you start growing, that's when you break out an independently scalable part of the system into a microservice

Respectfully disagree. At this point you start refactoring your monolith into components and actually look at performance measurements via tracing.

Do not do microservices when you're growing (or ever, most of the time).

And by the love of god, don't split your data. Data is much more complex to manage than code.

At several places, the biggest challenge to engineering velocity and the ability to innovate is that models are joined in interesting ways that prevent them from being decoupled. A user model tied to the package model preventing independent scaling of either. Every module reaching into all the tables. Joins, joins, and joins. Vast scans. The queries become inefficient and even the mighty postgres slows down at 2k rps. It is just too tempting to reach into another team's datastore. Now the two teams are in lockstep and can't alter their own datastore because others have assumptions on how it is stored and they access it directly.

At SendGrid, we had an few tables to deal with IP management. Over time, the like 5 tables were in use by 15+ services owned by different teams with competing priorities. When we finally had to scale the database, it took over three quarters of working with other teams while we supported both legacy and the new hotness.

It is hard to see early on where whole teams can sprout up and have domain ownership and to know which areas will be common/platform-like for other teams. But as soon as two services share a table, you should see a train coming at you.

There is severe hindsight bias in play here. How many tables weren't problematic despite all those joins and just chugged along scaling fine? How many startups were bogged down by unnecessary complexity or trying to keep a semblance of consistency instead of shipping features and just didn't reach the scale when performance becomes a problem?

Joins are good, actually, and normalisation should be the default. It's much easier to keep things consistent by default rather than trying to make them somewhat consistent afterwards. When something becomes a problem there is a bunch of tools (from denormalisation or materialised views to specialised datastores) to deal with that, but before that it's just premature optimisation.

> And by the love of god, don't split your data.

I think people feel that if they introduce a new data store, they won’t have to deal with the existing nearly unusable massive data store.

Of course that just exacerbates the problem.

Splitting data is just a hardcore way of splitting the responsibility for the data.

You can't have 42 different classes directly poking the User-table for example. You need one clear location that has the responsibility for the data.

If you move the User-table to a different database schema, other places CANNOT touch it because they won't have access to it =)

And then the order table will need to set user ids as a foreign key and you will have two sources of truth lol

Will be fun when you decide that you want to have organizations to be linked to orders instead of users.

All this to say, data management and having more than one source of truth complicates your world a lot. The responsibility for writing the data has to be solved at the appropriate layer: only one entity writing it.

> And then the order table will need to set user ids as a foreign key and you will have two sources of truth lol

Why should your order manager service be tightly-coupled to your IAM service when the only responsibility it has is to store a single ID?

> And then the order table will need to set user ids as a foreign key and you will have two sources of truth

You might be databasing wrong if that results in two sources of truth.

> And by the love of god, don't split your data.

Most data problems can be fixed on the frontend with a few relatively simple graphql queries. /s

You are trying to get me going, aren't you?

There is a way to splitting data. The service which owns the data always does the writes and others who need reads on that data can store replicas. ofc the complication then will be in replicating data, but this will enable services to massively scale and eliminate SPOFs

The other part of this pattern you mention (replicating data) seems solved in many cases by the data warehouse patterns of the last few years. Stuff it all in Snowflake or BigQuery then readers can query as they see fit. Query engines like Trino can paper over data storage heterogeneity / breakdowns in centralization. I'm not a fan of the "lakehouse" terminology, but it is the thing.

There are downsides to coupling more loosely with data consumers, but it keeps service owners moving without wasting time vending data.

As long as you're not replicating the data model also which means you're now replicating the business logic that go along with the data model.

An ETL approach from the start will save you a lot of headache.

> Respectfully disagree. At this point you start refactoring your monolith into components and actually look at performance measurements via tracing.

No matter how hard you modularize and instrument your service, that won't give you regional deployments, which you absolutely need to have ti be able to drive down customer-facing latencies from many hundreds of milliseconds to a few tens of milliseconds.

How do you pull that off with a monolith when you have a global userbase?

Do the next simplest thing: shard geographically. If you really need to handle users transiting geos, that's still dramatically simpler than meegroserbusses.

Sharding is part of how we do it.

Wise words. They boil down to a very simple truth that was as accurate half a century ago as it is today:

Make things as simple as possible, and as complex as necessary.

I can always make something more complex than it is now. As you say, we can take out things from a monolith and make them into a service. It can be hard to do so, sure. But nowhere near as hard, as trying to get complexity OUT of a system once it's in. Everyone who ever tried to revert a bunch of microservices back into a Monolith knows exactly what I am talking about. It usually amounts to the same work as a ground-up rebuild.

> It's so much easier to start out with one codebase and one database, and that will scale for a while.

Lots of businesses don't even need to scale anyway; Netflix is a "high customer, low revenue per customer" type of business, but there's lots of "low customer, high revenue per customer" businesses too, perhaps even more than the first one. These are often the type of products where you could quite literally run production on your laptop if you wanted to.

At the last place I worked they built all this microservice bonanza for ... 300 customers... If they ever got a few thousand customers it would be quite successful, and tens of thousands would be hugely successful. What usually happens is that they don't spend that "25% of your engineering time just maintaining the platform", so the platform was impossible to run locally, and production was hanging together with duct tape.

(Aside: in a recent interview I was asked to design a system for "500 million concurrent users" – this is for a business that's mostly in the low-customer/high-revenue type. I still don't know if the test was to point out that 500M is an utterly bonkers number – about a 10% of the internet connected people on the planet – or that they really thought this was somehow a vaguely realistic number for a startup. I answered with "build the simplest thing possible, a monolith, and focus on features and making sure you're building something people want". I didn't get hired, so I guess they were somehow under the misapprehension that you need to design things for 500M users right from the start?)

Ah, it’s you!

You’ve got a hell of a resume. And been accidentally incredibly convincing in getting many people into many early messes.

Jokes aside, thanks for at least coming around to advise startups sanely.

I’d personally never advocate for microservices until at the scale of Netflix or Amazon or Reddit, and even then only with in-house expertise at your level. Otherwise it’s a nightmare.

Thanks for everything, especially your contributions of sanity.

Aww, thanks for the kind words. I apologize if I caused you any harm with my talks. I did in fact start out saying everyone should use microservices, but I pulled back as I saw how damaging that can be to a small startup, or even a large enterprise that doesn't actually need it.

We all make mistakes!

If you want a sobering look at the damage you've helped do, go look at job listings on a generic job site.

At least here in the UK almost all of them have microservices listed as a requirement.

Given that at best like 5% of companies would benefit from microservices, that's a terrible spot our industry is in.

And the more sober of us would say 5% is a huge exaggeration (unless you mean by headcount)

I wrote this 8 years ago: Microservices vs. "air-gapped" modules https://www.linkedin.com/pulse/maintainable-software-archite.... You can achieve the same end as a microservice using a module, by simply having lint rules that prevent you from importing other application level modules. This way the only possible comms interface is to pass events with a payload of primitive typed parameters.

An interesting idea. Sort of a good half way. But one issue I see is that you can still have a shared data store. That means you can accidentally (or intentionally) use the database to pass back-channel messages —- have one module store data and another read it.

That could lead to hard to find bugs. Did this ever come up for you?

The best "microservice" based system I've ever worked on was full ten years before the term had been invented. It had multiple services that read and wrote to the same database, but each used a common dynamically linked library that was the common interface to access that database (no naked SQL was allowed -- you had to #include <foo> and then do Foo* foo = Foo::findById("xxx"); etc. Worked remarkably well. The above common library was very carefully maintained and any addition had to be reviewed and approved. The approach seemed to work well.

That's the services model that Facebook uses and parts of Amazon use. A single data store surrounded by a bunch of small services with a prescribed data access layer.

That's really more of a monolith though. One of the key tenants of microservices is data separation -- each service has its own data store.

I think systems like this are prone to buckling under product pressure. If it takes a few minutes to override a lint rule it's bound to happen sooner or later. With a microservice system the amount of effort is genuinely higher to add cross-dependencies.

I think if we also "air-gap" (loosely) the dependencies, we get a typical monorepo in, say, JS or Golang? That is, a module in a monorepo is a special case of your airgapped modules?

Not exactly sure what you mean, but in my model, the modules all get built into a single runtime artifact/executable. In the case of monorepos as I understand them, the "mono"ness is in the version control system, but each code base builds to its own artifact or executable, which have to communicate with each other over a network interface.

I agree - microservices are a technical solution to a people problem. Stevey's seminal "Platforms Rant" (https://gist.github.com/chitchcock/1281611) touches on this - the reason why Dread Pirate Bezos mandated services was not because Kubernetes was cool[1], but because their teams were too interconnected, moving too slowly, and Conway's law was getting in the way.

Splitting into services siloed each team and allowed them to move independently and by side effect, faster. Not due to inherent properties of (micro) services but because one goes faster by removing the things that slow you down.

As a startup, you do not have this problem. You likely will _never_ have this problem as your default future state is 'dead'.

Do the simplest thing that can possibly work - build a monolith using tools you know that give you the ability to predictably, rapidly iterate on the product.

After you hit some semblance of product/market fit and need to scale, you can do that. Scaling is a solved problem. Premature scaling is not.

[1. This is a joke. Kubernetes wasn't even a thought in Google's eye at this point in history]*

> the reason why Dread Pirate Bezos mandated services was not because Kubernetes was cool[1], but because their teams were too interconnected, moving too slowly, and Conway's law was getting in the way.

Not quite. Amazon's problem was that they were experiencing too much impedance between teams, and some teams were even siloing themselves to the extent they were creating problems for everyone around them. Bezos' diktat was intended to break through a bunch of petty office politics bullshit that was dragging down the company and preventing teams from delivering their work. The diktat boiled down to basically ordering everyone to grant access to the service they owned and provided to external teams that need it, no exception, and would be held liable if they failed to provide it, intentionally or not

Most people don't understand the point of microservices. They look at the idea and are attracted by the power of modular interfaces, customizable scalability and independent deployment, without really thinking of the additional engineering overhead that are required to unleash these capabilities.

I've seen ex-Netflix software engineers taking jobs elsewhere and proposing microservices for systems that would receive little to no benefit from them. In practice, the implementation of microservices in these contexts become a costly solution looking for a problem.

Just chipping in with thoughts on DynamoDB, (although I have worked on much smaller scale systems)

I am a long term dev, done lots of SQL, but for the past few years I have been using DynamoDB, and I am using it for my new startup (So I rate it).

Cons - You have to be very aware of your query patterns, and not having ad-hoc queries is a pain.

Plus sides - With on demand billing, its free if you aren't using it - Built correctly, it will scale - No Schema upgrades (This one is massive for me)

On the last point, I really do appreciate not having to worry about keeping schemas upto date across all devs and environments.

We use quite a simple pattern of a table per entity, as opposed to single table design, because it allows us to just use the API at the highest level of abstraction, where you just write your objects to dynamoDB. (You can still do lower level requests to poke values and such like)

> No Schema upgrades (This one is massive for me)

At Amazon, relational databases are banned unless you get an explicit exemption from senior leadership. This is the primary reason why. Too many cases of schema upgrades causing outages.

The problem with DDB or other NoSQL applications, like you say, is how much you need to consider your query patterns. The last major project I worked on using DDB, we spent a couple of days just thinking through our query patterns so we could come up with the right database design and data structures. (We still believe it was the right choice, though.)

I wish you good luck with your redemption arc, every monolith counts.

>Microservices takes at least 25% of your engineering time just maintaining the platform. It's not worth it unless you can recoup that 25% in efficiency.

My take is, microservices multiply your operational problems so you need a really solid platform/infrastructure/developer experience team otherwise you'll introduce a bunch of new headaches. These include things like CI/CD (building, storing artifacts, testing artifacts, promoting artifacts between environments), observability (metrics, logs, distributed tracing, error monitoring), distributed systems quirks like cascading failures, release/change management and communication, service discovery/routing, automated infra provisioning for things like datastores. Unless you have a really good handle on all these pieces (which I think most startups don't), you end up in an operational nightmare spending tons of time trying to keep the system going.

Of course, the next step is throwing on 3rd party products to try to solve this which adds even more complexity. Throw in Datadog for observability, Istio service mesh for networking/traffic, ArgoCD/kustomize/Helm/Kubernetes to manage all this infra, etc etc

How would you split the total overhead between monolith and monorepo?

(sorry this ran away with me)

A dumb example is that if I start with my single codebase running on a single server, I am likely to have a single git repo (foo).

Then I have a genius idea and put all the email handling code into foo.mail and soon I have foo.web and foo.payments.

All is fine as long as I am just checking out HEAD each time. The code running in the runtime is still one big set of code.

If I get creative and put in load balancers it's still a monolith.

But if I split out the web servers from the email servers, then I kind of start to see microservices appear.

I am trying not to be pedantic, but I am truly interested in experienced views on where the pain really starts to appear.

At this point (server to server comms), I should look at mTLS, and centralised logging and all the good stuff to manage microservices.

But how much pain was there before?

What if I was a large company and so hired a dev or two per repo (you know to get that 9 women one month effect). Co-ordinating multiple devs over different repos, even with one monolithic runtime, is painful (experience tells me).

So I am interested in where the break points are, and whether there are easier paths up the mountain?

Splitting out the webserver (assuming it is the main entrypoint for users) seems more like an infrastructure choice than application architecture and having an independent email-sending-service looks more like replacing a third-party offer (like turboSMTP) with an in-house service.

I do not think that this is what people mean by microservices.

I get that. I was trying to describe a small path from monolith to microservice without inventing yet another student and teacher database. But the even the step you mention matters. going to a third party service is very similar to reaching out to your own internal service. Make it a "customer" service.

Is the pain in microservices (or APIs and mTLS) or is the pain in managing a team that now has to do customer stuff only ? or email stuff only?

> Make it a "customer" service.

Common path to pain right there — services based on nouns and entities rather than verbs and business processes.

Having a single "customer" service means anything involving customer data has to talk to that single service. If that service has a problem, every other service stops working. You basically have a tightly coupled distributed monolith, so network boundaries instead of function calls between modules.

Think instead of splitting that data to separate services around business processes like billing, shipping, outbound marketing, whatever. Each service owns the slice of customer data required for its processes and can run fully autonomously.

At Netflix we had both noun and verb services. We had the "play movie" service, which was very much verb, as well as the listing service for example. But we also had the subscriber service, which was the source of truth for any account data, and the catalog service, which held all the info about the movies.

Yes, subscriber was a Tier 0 service and required a strong reliability and scaling story. They had more rigorous deployment setup and were usually over-scaled for traffic. But they also vended a library that services used to connect to them. That library had intelligent caching and fallbacks, so that even if the service was down, in a lot of cases the system could keep running. And the catalog was the same.

Microservices is very intertwined with how your engineering teams are built. We had a team that worked on subscriber data, one that worked on catalog data, one that worked on making movies play, one that built listings. So we had services to match.

Thank you. I sometimes feel microservices are an evolution of Conways law. How a company sees its internal organisation is how it will want to arrange its microservices and vice versa.

I remember trying to build out a "barium bullet" through very complex intertwined systems - following known customer data and looking for their footprints in all linked systems. It gave some insights on how to redesign systems - like someone else said here, removing complexity is ridiculously hard.

That seems like a great approach for Netflix and investing in a resilient, redundant Tier 0 service like that was likely worth it there

Most orgs don't operate on that scale though and resources would likely be better spent elsewhere

> Microservices is very intertwined with how your engineering teams are built.

I wish more people understood this. Microservices are much more about scaling your organization than about scaling technology. At a certain size, teams can't collaborate effectively enough and splitting out different services for each team makes a lot of sense

…until you need to market to people who didn't get a shipping last month, delay shipping until at least one recharge attempt, or ship before charging for customers you're reasonably sure will pay anyway.

Business changes, and putting rigid firewalls in place without seeing an established pattern of change and without established need for that is a bad idea.

One global database buys you a lot (easy data access, easy transaction consistency, simple locks, even transactional notifications), and you can take it very far performance wise, throwing that away just in case you might need more performance later seems unwise.

> …until you need to market to people who didn't get a shipping last month, delay shipping until at least one recharge attempt, or ship before charging for customers you're reasonably sure will pay anyway.

These are all very simple things to handle when services encode processes instead of entities

> Business changes

Yes, and then you throw away the code for that outdated process and encode the new one.

> One global database buys you a lot

Absolutely, this is the correct choice for probably 95+% of companies. I will always argue in favour of a monolith and single database.

If someone is dead set on microservices though, they'll avoid a world of pain by actually designing them properly

I honestly believe that doing small services is long term better approach than a monolith. But as you I strongly believe it's not where you want to start.

But I do not agree on is more time consuming than the other. It's just time spent on different matters. For the sake of money spent, I have not seen any data on how a monolith outperforms small services. For maintaining the platform, I think 25% sounds a bit high.

Money spent should be measured in many different aspects. One is definitely productivity. And I have seen more stale monoliths than I have seen stale small services. Whether it's better or not is not up to me to judge. I just know what I prefer.

Having been exposed to small service architecture where it has been working really well and very poor, there are few things that stand out.

- Conquer and divide (your monolith over time). - Responsibility boundaries are easier to cope with for developers in a small service since they often don't have generic and yagni abstractions. - Less code to comprehend for a small service and the cognitive load decreases. - Build time, test time, deployment time. - Should lean up against a direct business measurement and value. - Group chatting services into one.

I have yet to see a monolith that over time is not really deteriorating, but I haven't worked with SO or Shopify. We could also argue that Cobol and Fortran is still of good use but time has also changed leaving that style of system development exposed as old and dusty.

But like any software development occurrence, it takes responsibility, mandate and proper leadership to get things done in a decent manner. So if you start by making 3 services that has to chat to find a user profile you probably don't know what you're doing. And 25 people (the Threads image) is not a small team IMO.

Good luck

My go to page about this is http://widgetsandshit.com/teddziuba/2008/04/im-going-to-scal... which perfectly summarizes the problem AND solution

The issue isn't blindly adopting microservices or a monolith as a religion or bikeshedding thereabouts.

Architectural and ops concerns lead to dividing an overall service into sufficient and necessary units of abstraction. To efficiently run more than 1 service leads to standardization of automation of the concerns of application platform infrastructure:

- stateful data backup, protection recovery

- configuration management

- OS security

- authentication, authorization, audit, encryption, and identity management

- monitoring

- analytics

- rate limiting

- A/B, etc. feature tests

- sharded deployment

Chopping up 1 service into many more services doesn't make the above concerns go away. Neither does collapsing many into 1.

The internal pieces (and often interfaces between business units) need to be broken down into decoupled units of abstraction. If people want to call that "microservices" or a "monolith", it's kind of irrelevant. Containers of abstraction should serve a purpose rather than hinder it.

> And then, when you start growing, that's when you break out an independently scalable part of the system into a microservice (with it's own data store!).

Wasn't this the rule of thumb for microservices from the very start? That the main driving force is to ensure separate teams work independently on stand-alone projects they own exclusively? Scaling tends to only become an issue when the project grows considerably.

Yes. But that idea got lost in the fray. Even I tried to build a system from scratch with microservices at one point.

> Yes. But that idea got lost in the fray.

I don't think so. One of the very first things that people learn when onboarding into microservices is the microservices tax, whose corollary is don't onboard onto microservices if there isn't a clear significant win that justifies all the tradeoffs.

Moreover, the main selling point of microservices is organizational. That's also right in the first lesson on microservices. If you're not working in a large team who owns a large project and wants to lower managerial overhead by splitting up the team, you need to try very hard to find a justification to peel off specific responsibilities from a service into an independent microservice. That's microservices 101.

Those who jump onto the microservicss bandwagon without having a concrete justification have only themselves to blame, and not any particular architectural style. They are making the piss poor decisions and blaming abstract concepts for their own mistakes.

> break out an independently scalable part of the system into a microservice … The one that may need to scale independently, or that you want to be able to deploy separately.

How often would you estimate that security concerns are a valid reason to isolate functionality into a microservice? Separation of concerns, reduced attack surface and blast radius, etc

Wise. Every time I’ve started a project trying to make an amazing platform with everything segregated with micro services etc. I’ve spent so much more time on the _platform_ than on the actual product.

Now I go with SQLite and some basic python script and pivot from there. Ironically that’s how I used to do it before the micro service fad.

> And did you know that you can deploy a monolith to Lambda and still get all the benefits of Lambda without building services

Could I get more info on this? All I've found so far is that it's an anti-pattern. Isn't the startup time a killer?

> Isn't the startup time a killer?

It depends on your programming language and how many libraries you are importing. If you use Python you can get startup times less than 200ms. Also, if your app is active, most startups won't be cold (Lambda reuses the VM a bunch of times before recycling it if it's staying active). You can also add a health check API and just poke that once a minute to keep it warm if you want (at the cost of increased invocations).

But yes, if you're doing something that requires a lot of setup, then this pattern isn't for you. But it turns out most web apps are simple enough it can be done this way, especially if you're already using a pattern where the frontend is doing a lot of the heavy lifting and your API is mostly just marshaling data from the client to the database with some security and small logic in between.

Absolutely agree, I've seen far too many companies spend far too much time working out the infrastructural relationships of microservices. Spending very little time working on the actual applications needs itself.

The problem is not with the devs but with the investors I believe. They're looking for unicorns, unicorns attract million of customers, so they need to work at scale, hence microservices.

Well, in the previous cycle it was clear that the investors were looking for buzzwords and marketing hype because they often don't understand and frankly dont care - they just wanted to make you look good enough to sell to the next guy, as they were all assuming you were unprofitable in and unprofitable out.

They all just want a story that you are going to be the next google so they can sell that story to the next guy.

Will that hold in the non zero interest rate world? The AI hype doesn't make me think the world has changed that much.

make up a new word for monolith that sounds sexy instead of big, bulky and scary.

Something like "uniform-services" or "harmonized-services". Vertically-distributed harmonized services.

> And did you know that you can deploy a monolith to Lambda and still get all the benefits of Lambda without building services

I did not know. Does anyone have pointers or examples on this?

As an example, you'd write your entire API as a flask app, and then deploy that app to Lambda. Then send all requests to that one Lambda. As long as your startup time is quick (and your datastore is elsewhere, like in DynamoDB) then it will work great for quite a while. Lambda will basically run your app enough times to handle all the requests.

You have to be careful when you design it such that you don't rely on subsequent requests coming to the same machine, but you can also design it so that if they do come to the same machine it will work, using a tiered cache.

I saved url of your comment into my favourites, if I need to convince people in a future.

Obligatory "PostgreSQL when it's not your job" share. It'll get many shops pretty damn far imo

https://www.pgcon.org/2016/schedule/attachments/437_not-your...

Of course it is even better to build a distributed monolith...

Reply on news.ycombinator.com