Hacker News Reader

Comment by xyst | Hacker News Reader

Of course there’s no mention of performance loss or gain after migration.

I remember when microservices architecture was the latest hot trend that came off the presses. Small and big firms were racing to redesign/reimplement apps. But most forgot they weren’t Google/Netflix/Facebook.

I remember end user experience ended up being _worse_ after the implementation. There was a saturation point where a single micro service called by all of the other micro services would cause complete system meltdown. There was also the case of an “accidental” dependency loop (S1 -> S2 -> S3 -> S1). Company didn’t have an easy way to trace logs across different services (way before distributed tracing was a thing). Turns out only a specific condition would trigger the dependency loop (maybe, 1 in 100 requests?).

Good times. Also, job safety.

View on HN

As often, Grug has a great line about microservices:

    grug wonder why big brain take hardest problem, factoring system correctly, and introduce network call too

    seem very confusing to grug

https://grugbrain.dev/#grug-on-microservices

Thanks for the link to this awesome page!

Thanks for that link, I genuinely laughed out loud while reading some of those points! Love the presentation and what a wonderful reality check I can’t agree with more.

Extremely slow times - from development to production, backend to frontend. Depending on how bad things are, you might catch the microservice guys complaining over microseconds from a team downstream, in front of a FE dev who’s spent his week optimizing the monotonically-increasing JS bundles with code splitting heuristics.

Of course, it was because the client app recently went over the 100MB JS budget. Which they decided to make because the last time that happened, customers abroad reported seeing “white screens”. International conversion dropped sharply not long after that.

It’s pretty silly. So ya, good times indeed. Time to learn k8s.

Even if the microservices platform is running at 1% capacity, it's guaranteed to have worse performance than almost any monolith architecture.

It's very rare to meet a developer who has even the vaguest notion of what an RPC call costs in terms of microseconds.

Fewer still that know about issues such as head-of-line blocking, the effects of load balancer modes such as hash versus round-robin, or the CPU overheads of protocol ser/des.

If you have an architecture that involves about 5 hops combined with sidecars, envoys, reverse proxies, and multiple zones you're almost certainly spending 99% to 99.9% of the wall clock time just waiting. The useful compute time can rapidly start to approach zero.

This is how you end up with apps like Jira taking a solid minute to show an empty form.

Are you talking about cloud Jira? I use it daily and it's very quick, even search results appear immediately...

Have you ever used an old ticketing system like remedy? Ugly as sin, but screens appear instantly.

I think Web apps have been around so long now people have forgotten how unresponsive things are vs old 2 tier stuff.

There's a definite vibe these days of "this is how it's always been" when it really hasn't.

The SPA Jira cloud is faster than anything server rendered for me, my connection is shit. In Jira I can at least move to static forms quickly and I'm not downloading the entire page on every move.

I’ve used Remedy in three different shops, and the experience varied dramatically. The entry screen might have popped instantly, but RPC timeouts on submission were common. Tab order for controls was the order they were added, not position on the screen. Remedy could be pleasant, but it was very dependent on a competent admin to set up and maintain it.

Maybe you're being honest, but you're using a throwaway account and I use Cloud Jira every day. It's slow and bloated and drives me crazy.

Has a LOT to do with how it's configured and what plugins are installed.

It really doesn't. This is the excuse trotted out by Atlassian staff when defending their products in public forums, essentially "corporate propaganda". They have a history of gaslighting users, either telling them to disregard the evidence of their own lying eyes, or that it's their own fault for using the product wrong somehow.

I tested the Jira cloud service with a new, blank account. Zero data, zero customisations, zero security rules. Empty.

Almost all basic operations took tens of seconds to run, even when run repeatedly to warm up any internal caches. Opening a new issue ticket form was especially bad, taking nearly a minute.

Other Atlassian excuses included: corporate web proxy servers (I have none), slow Internet (gigabit fibre), slow PCs (gaming laptop on "high performance" settings), browser security plugins (none), etc...

> Opening a new issue ticket form was especially bad, taking nearly a minute.

At that point, something mustve been wrong with your instance. I'd never call jira fast, but the new ticked dialog on a unconfigured instance opens within <10s (which is absolutely horrendous performance to be clear. Anything more then 200-500ms is.)

Cloud Jira is notably slower than on-prem Jira, which takes on the order of 10 seconds like you said.

That does not mirror my own experience. And it's very easy to validate. Just create a free jira cloud instance, takes about 1 minute ( https://www.atlassian.com/software/jira/try ) and click new issue.

it's open within 1-2 sec (which is still bad performance, objectively speaking. It's an empty instance after all and already >1s )

Ah, there's a new UI now. The last time I tested this, the entire look & feel was different and everything was in slow motion.

It's still sluggish compared to a desktop app from the 1990s, but it's much faster than just a couple of years ago.

This is just not true. The create new issue form appears nearly immediately. I have created two tickets right now - in less than a minute including the writing of few sentences.

Atlassian themselves don't use JIRA Cloud. They use the datacenter edition (on-premise) for their public bug tracker, and it's sooooo much faster than the Cloud version: https://jira.atlassian.com/browse/

With cloud Jira you get thrown on a shared instance with no control over who you're sharing with, so it's random whether yours will be fast or extremely slow.

I’m starting to think that these people praising Jira are just part of an architected PR campaign that tries to deny what’s evident to the end users: Jira is slow, bloated in many cases almost unusable.

> It's very rare to meet a developer who has even the vaguest notion of what an RPC call costs in terms of microseconds.

To be fair, small time units are difficult to internalize. Just look at what happens when someone finds out that it takes tens of nanoseconds to call a C function in Go (gc). They regularly conclude that it's completely unusable, and not just in a tight loop with an unfathomable number of calls, but even for a single call in their program that runs once per day. You can flat out tell another developer exactly how many microseconds the RPC is going to add and they still aren't apt to get it.

It is not rare to find developers who understand that RPC has a higher cost than a local function, though, and with enough understanding of that to know that there could be a problem if overused. Where they often fall down, however, is when the tools and frameworks try to hide the complexity by making RPC look like a local function. It then becomes easy to miss that there is additional overheard to consider. Make the complexity explicit and you won't find many developers oblivious to it.

Those time cost need to contextualize with time budgets for each service. Without that, it is always somebody else's problem in a RPC world.

>Just look at what happens when someone finds out that it takes tens of nanoseconds to call a C function in Go (gc).

I'm not too familiar with Go but my default assumption is that it's just used as a convenient excuse to avoid learning how to do FFI.

> This is how you end up with apps like Jira taking a solid minute to show an empty form.

Nah. Jira was horrible and slow long before the microservice trend.

JIRA slowness usually involved under provisioned server resources in my experience

Nope. You could put it on a beefy machine with a beefy database and it would still crawl a lot of the time. It just had some combination of terrible database querying and general slowness. The whole generic inner-platform effect way it's constructed (e.g. having its own ticket query language that they presumably translate generically into SQL) was probably a big factor.

Not discounting your knowledge, but somehow that was never the case for me across multiple companies, with I think one exception and even that one wasn't as slow as people talk about, despite being the worst I ever encountered.

In fact, I'd argue that one of the biggest annoyances I had in last job was that I actually missed JIRA, or rather it's glorious issue list view with its keyboard shortcuts.

It's actually worse than what you said. In 2024 the network is the only resource we can't upgrade on demand. There are physical limits we can't change. (I.e., there are only so many wires connecting your machines, and any significant upgrade involves building your own data center.)

So really eventually we'll all be optimizing around network interfaces as the bottleneck.

cough speed of light cough

Why would moving to Kubernered from ECS introduce performance issues?

They already has their architecture, largely, and just moved it over to K8s.

They even mention they aren’t a micro service company.

> There was a saturation point where a single micro service called by all of the other micro services would cause complete system meltdown.

Yep - saw that at a company recently - something in AWS was running a little slower than usual which cascaded to cause massive failures. Dozens of people were trying to get to the bottom of it, it mysteriously fixed itself and no one could offer any good explanation.

Any company that doesn’t have some form of distributed tracing in this day and age is acting with pure negligence IMO. Literally flying blind.

Distributed tracing?

Is that the same as what Elixir/Erlang call supervision trees?

Likely about Opentelemetry

Oh, come on! Just make sure your architecture is sound. If you need to run an expensive data analysis cluster connected to massive streams of collected call information to see that you have loops on your architecture, then you have a bigger issue.

I don't know if you're being sarcastic, which if you are: heh.

But to the point, if you're going to build a distributed system, you need tools to track problems across the distributed system that also works across teams. A poorly performing service could be caused by up/downstream components and doing that without some kind of tracing is hard even if your stack is linear.

The same is true for a giant monolithic app, but the sophisticated tools are just different.

Solution: don't build a distributed system. Just have a computer somewhere running .NET or Java or something. If you really want data integrity and safety, just make the data layer distributed.

There's very little reason to distribute application code. It's very, very rare that the limiting factor in an application is compute. Typically, it's the data layer, which you can change independently of your application.

I have yet to personally see an application where distribution of its parts was beneficial. For most applications a boring monolith works totally fine.

I'm sure it exists when the problem itself is distributed. For example, I can imagine something like YouTube would require a complex distributed system.

But I think very few problems fit into that archetype. Instead, people build distributed systems for reliability and integrity. But it's overkill, because you bring all the baggage and complexity of distributed computing. This area is incredibly difficult. I view it similar to parallelism. If you can avoid it for your problem, then avoid it. If you really can't, then take a less complex approach. There's no reason to jump to "scale to X threads and every thread is unaware of where it's running" type solutions, because those are complex.

I see this in Android. Every few years (sometimes multiple times in a year) a new arch. becomes the fad and every TDH dev starts hankering for one way or the other on the lines "why are you not doing X..". Problem is, at immature firms (specially startups) the director and senior manager level leaders happily agree for the re-write i.e re-archs because they usually leave every season for the next place and they get to talk about that new thing which probably didn't last beyond their stay, or might have conked off before that.

And the "test cases" porn! Goodness! People want "coverage" and that's it. It's a box to tick, irrespective of whether those test cases and they way those are written are actually meaningful in anyway. There are more like "let's have dependency injection everywhere" charade.

The best part is when it’s all architected to depend on something that becomes essentially a single point of failure, like Kafka.

Not saying you're wrong, but what is your grand plan ? I've never seen anything perfect.

Shhh, you're ruining the party!

Isn't this somewhat better, at least when it fails it's in a single place?

As someone using Kafka, I'd like to know what the (good) alternatives are if you have suggestions.

It really depends on what your application is.

Where I'm at, most of Kafka usage adds nothing of note and could be replaced with a rest service. It sounds good that Kafka makes everything execute in order, but honestly just making requests block does the same thing.

At least then I could autoscale, which Kafka prevents.

NATS JetStream seemed to support horizontal scaling (either hierarchical via leaf nodes or a flat RAFT quorum) and back pressure when I played with it.

I found it easy to get up and running, even as a RAFT cluster, but I have not tried to use JetStream mode heavily yet.

At least Kafka can be properly master-master HA. How people ever got away with building massively redundant fault-tolerant applications that were completely dependent on a single SQL server, I'll never understand.

I think because it works pretty close to 100 percent of the time with only the most basic maintenance and care (like making sure you don't run out of disk space and keeping up with security updates). You can go amazingly far with this, and adding read only replicas gets you a lot further with little extra effort.

stack overflow has no problem serving entire planet from just four SQL Servers (https://nickcraver.com/blog/2016/02/17/stack-overflow-the-ar...)

There is really nothing wrong with a large vertically scaled up SQL server. You need to be either really really large large scale - or really really UNSKILLED in sql as to keep your relational model and working set in SQL so bad that you reach its limits

>or really really UNSKILLED in sql as to keep your relational model and working set in SQL so bad that you reach its limits

Sadly that's the case at my current job. Zero thought put into table design, zero effort into even formatting our stored procedures in a remotely readable way, zero attempts to cache data on the application side even when it's glaringly obvious. We actually brought in a consultant to diagnose our SQL Server performance issues (I'm sure we paid a small fortune for that) and the DB team and all of the other higher ups capable of actually enforcing change rejected every last one of his suggestions.

> There is really nothing wrong with a large vertically scaled up SQL server.

Large or small, it's always going to be a single point of failure. All hardware is fallible, especially if we're talking about a commodity box rather than a mainframe or something.

> How people ever got away with building massively redundant fault-tolerant applications that were completely dependent on a single SQL server, I'll never understand.

It works, with a lower cognitive burden than that of horizontally scaling.

For the loading concern (i.e. is this enough to handle the load):

For most businesses, being able to serve 20k concurrent requests is way more than they need anyway: an internal app used by 500k users typically has fewer than 20k concurrent requests in flight at peak.

A cheap VPS running PostgreSQL can easily handle that.[1]

For the "if something breaks" concern:

Each "fault-tolerance" criteria added adds some cost. At some point the cost of being resistant to errors exceeds the cost of downtime. The mechanisms to reduce downtime when the single large SQL server shits the bed (failovers, RO followers, whatever) can reduce that downtime to mere minutes.

What is the benefit to removing 3 minutes of downtime? $100? $1k? $100k? $1m? The business will have to decide what those 3 minutes are worth, and if that worth exceeds the cost of using something other than a single large SQL server.

Until and unless you reach the load and downtime-cost of Google, Amazon, Twitter, FB, Netflix, etc, you're simply prematurely optimising for a scenario that, even in the businesses best-case projections, might never exist.

The best thing to do, TBH, is ask the business for their best-case projections and build to handle 90% of that.

[1] An expensive VPS running PostgreSQL can handle a lot more than you think.

> Each "fault-tolerance" criteria added adds some cost. At some point the cost of being resistant to errors exceeds the cost of downtime.

Not to forget: those costs are not just in money and time, but also in complexity. And added complexity comes with its own downtime risks. It's not that uncommon for systems to go down due to problems with mechanisms or components that would not exist in a simpler, "not fault tolerant" system.

The business can try to decide what those 3min are worth, but ultimately the customers vote by either staying or leaving that service.

> The business can try to decide what those 3min are worth, but ultimately the customers vote by either staying or leaving that service.

That's still a business decision.

Customers don't vote with their feet based on what tech stack the business chose, they vote based on a range of other factors, few, if none, of which are related to 3m of downtime.

There are little to no services I know off that would lose customers over 3m of downtime per week.

IOW, 3m of downtime is mostly an imaginary problem.

That's really a too broad generalization.

Services that people might leave, because of downtime are for example a git hoster, or a password manager. When people cannot push their commits and this happens multiple times, they may leave for another git hoster. I have seen this very example when gitlab was less stable and often unreachable for a few minutes. When people need some credentials, but cannot reach their online password manager, they cannot work. They cannot trust that service to be available in critical moments. Not being able to access your credentials leaves a very bad impression. Some will look for more reliable ways of storing their credentials.

The user experience of "often unreachable" means way more than 3m per week in practice.

Why does a password manager needs to be online. I understand the need for synchronization, but being exclusively online is a very bad decision. And git synchronization is basically ssh, and if you mess that up on a regular basis, you have no job being in business in the first place. These are examples, but there's a few things that do not need to be online unless your computer is a thin client or you don't trust it at all.

>> What is the benefit to removing 3 minutes of downtime?

> The business can try to decide what those 3min are worth, but ultimately the customers vote by either staying or leaving that service.

What do you think the business is doing when it evaluates what 3 minutes are worth?

There is no "the business". Businesses do all kinds of f'ed up things and lie to themselves all the time as well.

I don't understand, what people are arguing about here. Are we really arguing about customers making their own choice? Since that is all I stated. The business can jump up and down all it wants, if the customers decide to leave. Is that not very clear?

> The business can jump up and down all it wants, if the customers decide to leave.

I think the point is that, for a few minutes of downtime, businesses lose so little customers that it's not worth avoiding that downtime.

Just now, we had a 5m period where disney+ stopped responding. We aren't going to cut off our toddler from peppa big and bluey for 5m of downtime per day, nevermind per week.

You appeared to be under the impression that 3m downtime/week is enough to make people leave. This is simply not true, especially for internet services where the users are conditioned to simply wait.

I think people are arguing that what you say makes no sense as a counter argument. Of course customers make their choice. THAT IS WHAT BUSINESSES ARE CALCULATING. How many customers do we lose as a result of downtime, how much is that worth in $ and how many $ do we need to spend to not suffer that downtime? If the projected loss is less than the cost of fixing the downtime, let the customers go.

You saying “the customer can decide to leave” is not a counter to this at all. It’s just a weird way of saying what everyone else is but framing it as a counter argument to what is being said. Which it simply isn’t.

True, but what people should understand about databases is they're incredibly mature software. They don't fail, they just don't. It's not like the software we're used to using where "whoopsie! Something broke!" is common.

I've never, in my life, seen an error in SQL Server related to SQL Server. It's always been me, the app code developer.

Now, to be fair, the server itself or the hardware CAN fail. But having active/passive database configurations is simple, tried and tested.

And the server itself can be very resilient if you run something like debian or freebsd. Even on arch, I've seen things fails rarely unless it's fringe/proprietary code (bluetooth, nvidia, the browser and 3d accelerated graphics,...). That presumes you will use boring tech which are heavily tested by people around the world, not something "new" and "hyped" which is still on 0.x

I agree 100%. Unfortunately my company is pretty tied to windows and windows server, which is a pain. Upgrading and sysadmin-type work is still very manual and there's a lot of room for human error.

I wish we would use something like Debian and take advantage of tech like systemd. But alas, we're still using COM and Windows Services and we still need to remote desktop in and click around on random GUIs to get stuff to work.

Luckily, SQL Server itself is very stable and reliable. But even SQL Server runs on Linux.

> For most businesses, being able to serve 20k concurrent requests is way more than they need anyway: an internal app used by 500k

This is a very simple distinction and I am not sure why is it not understood

For some reason people design public apps the same as internal apps

The largest companies employ circa 1 million people - that’s Walmart, Amazon, etc. most giants, like Shell, etc. companies have ~ 100k tops. That can be handled by 1 beefy server.

Successful consumer facing apps are hundred millions to billion. It’s 3 orders of magnitude difference

I have seen a company with 5k employees invest into mega-scalable microservice event driven architecture and I was was thinking - I hope they realise what they are doing and it’s just CV-driven development

> Each "fault-tolerance" criteria added adds some cost. At some point the cost of being resistant to errors exceeds the cost of downtime.

Agreed, but the cost/benefit should be the same at every level in the stack. I get it when people build a single-server system. I'm just baffled that so many people insist they need a load balancer and multiple instances of their application so that there's no single point of failure (which is not free by any means), but then run them all off of a single SQL server.

Because people are beholden to costs and it's often out of our hands when to spend money on redundancy (or opportunity costs elsewhere).

It's less true today when redundancy is baked into SaS products (like AWS Aurora, where even if you have a single database instance, it's easy to spin up a replacement one if the hardware on the one running fails).

Ow yeah, I am looking at that problem right now

I'm sorry,but I can't tell if you're being serious or not since you commented without qualification.

One of the most stable system archictectures I've built was on Kafka AND it was running with microservices managed by teams across multiple geographies and time zones. It was one of the most reliable systems in the bank. There are situations where it isn't appropriate, which can be said for most tech e.g( K8S vs ECS vs Nomad vs bare metal)

Every system has failure characteristics. Kafka's is defined as Consistent and Available and your system architecture needs to take that into consideration. Also the transactionality of tasks across multiple services and process boundaries is important.

Let's not pretend that kubernetes (or the tech of the day) is at fault while completely ignoring the complex architectural considerations that are being juggled

Basically because people end up engineering their microservices as a shim to funnel data into the “magical black hole”

From my experience most microservices aren’t engineered to handle back pressure. If there is a sudden upsurge in traffic or data the Kafka cluster is expected to absorb all of the throughput. If the cluster starts having IO issues then literally everything in your “distributed” application is now slowly failing until the consumers/brokers can catch up.

This is a very fad driven industry. One of the things you earn after being in it for a long time is intuition for spotting fads and gratuitous complexity traps.

I think that aspect is indirectly covered, as one of the main motivation was to get on a popular platform that helps hiring.

I agree on how it's technically a waste of time to pursue fads, but it's also a huge PITA to have a platform that good engineers actively try to avoid, as their careers would stagnate (even as they themselves know that it's half a fad)

I avoid working at organisations with NIH syndrome - if they are below a certain size (i.e. they lack a standing dev-eng team to support their homegrown K8s "equivalent"). Extra red flags if the said homegrown-system was developed by that guy[1] who's ostensibly a genius and has very strong opinions about his system. Give me k8s' YAML-hell any day instead, at least that bloat has transferable skills, amd I can actually Google common resolutions.

1. Has been at org for so long, management condones them flaunting the rules, like pushing straight to prod. Hates the "inefficiency" of open source platforms and purpose-built something "suitable for the company" by themselves, no documentation you have to ask them to fix issues because they don't accept code or suggestions from others. The DSL they developed is inconsistent and has no parser/linter.

yeah. If you think Kubernetes is too complicated, the flip side of that is someone built the simpler thing, but then unfortunately it grew and grew, and now you've got this mess of a system. you could have just used a hosted k8s or k3s system from the start instead of reinventing the wheel.

absoutely start as simple as you can, but plan to move to a hosted kube something asap instead of writing your own base images, unless that's a differentiator for your company.

1. You have to constantly learn to keep up!

2. Fad-driven

I wonder why I don’t often see (1) critiqued on the basis of (2).

There’s definitely a connection. Some of the change is improvement, like memory safe languages, but a ton of it is fads and needless complexity invented to provide a reason for some business or consultancy or group in a FAANG to exist. The rest of the industry cargo cults the big companies.

> There was also the case of an “accidental” dependency loop (S1 -> S2 -> S3 -> S1).

The classic dependency loop example that you thought will never encounter again for the rest of your life after OS class

This article specifically mentions that they are not running microservices and has pretty clearly defined motivations for making the migration.

I imagine because the article mentions:

> More broadly, we’re not a microservices company, and we don’t plan to become one

It seems the "micro" implies that services are separated by high level business terms, like "payment" or "inventory" with each having their own databases instead of computational terms like "storage", "load balancing" or "preprocessing" etc.

Is this generally correct? How well is this term defined?

If yes, then I'm not surprised this type of design has become a target for frustration. State is smeared across system, which implies a lot of messaging and arbitrary connections between services.

That type of design is useful if you are an application platform (or similar) where you have no say in what the individual entities are, and actually have no idea what they will be.

But if you have the birds-eye view and implement all of it, then why would you do that?

>there’s no mention of performance loss or gain after migration

to illustrate performance cost i usually ask people what ping they have say from one component/pod/service to another and to compare that value to what ping they'd get between 2 linux boxes sitting on that gorgeous 10Gb/40Gb/100Gb or even 1000Gb network that they are running their modern microservices architecture over.

And if it was a single application, how would that solved it? You still would have that loop, no?

Personally, i think, it does not have to be hundreds of microservices, basically each function a service. But i see it more as the web in the internet. Things are sometimes not reachable or overloaded. I think that is normal life.

> single micro service called by all of the other micro services

So they didn't do microservices correctly. Big surprise.

I mean, that pattern is pretty common in the micro service world. Services for things like authz, locking, logging/tracing, etc. are often centralized SPOFs.

There are certainly ways to mitigate the SPOFiness of each of those cases, but that doesn’t make having them an antipattern.

One of the points of a microservice architecture (on k8s or otherwise) is that you can easily horizontally scale the component that was under pressure without having to scale out a monolithic application... that just sounds like people being clueless, not a failure of microservice architecture...

Or cost increase or decrease!

Reply on news.ycombinator.com