Making a system highly available isn't just about adding redundancy | Hacker News Reader

Making a system highly available isn't just about adding redundancy

meesho.io

32 pointsshsethi13 comments

Reading this, I can’t help but feel that things have got more complicated than perhaps they need to be. Microservices boundaries make most sense to me when they’re aligned with team boundaries, otherwise it just feels like making more work with little or no apparent gain.

> Microservices boundaries make most sense to me when they’re aligned with team boundaries, otherwise it just feels like making more work with little or no apparent gain.

Nothing in the article suggests how services were designed, and I would be surprised if each of the core services mentioned in the article weren't managed by independent teams.

Also, I don't think your concerns are well founded. Think about functionality implemented with function-as-a-service backends. Do you think your architecture is less complex to maintain and operate if you suddenly decide to move all event handlers into a monolith?

Are you going to rebuild your micro service every time there's a reorg?

It's nice as a SWE to own a specific service, but I think it leads to suboptimal tech stacks and adds latency.

> Are you going to rebuild your micro service every time there's a reorg?

I feel like the answer there might be "sometimes", not as something that you'd expect to be done, but something that arises from multiple teams now sharing a service and having some friction due to that. I can conceivably imagine those two teams wanting to split off their individual parts of the functionality to be separate.

With any sort of system, when there are multiple components involved, having clear ownership and being able to iterate quickly can be pretty good - for example, if there's a system that handles the management of all sorts of binary data (file uploads, metadata storage for those) then having one team be responsible for it but other teams just using it can be a good setup.

I would argue that you don't always need to chop up your domain into microservices (depending on the size of your domain), but definitely look into doing that across various technological boundaries (a team for your web API, your auth, scheduled processes and messaging, handling object storage etc.). If you do that too much, though, it's going to become a bit of a mess of granularity.

and what better way to spark joy on the new team than the opportunity to leave all the tech debt behind with a clean rewrite (in rust!)

only half joking.

Regardless of the language or the trendy tech used, people seem to enjoy doing rewrites way more than working with old codebases, especially if their new solution doesn't need to be 1:1 with the old one and therefore they can justify no longer needing to work around certain edge cases. That longing can't really be helped, though.

Honestly, as long as the second system effect doesn't ruin everything and you still make the new codebase better than the old one (i.e. it's small enough to actually be rewritten and testable enough to make sure that it actually works, with the abstractions that you now know you need and have solidified enough to be the right ones, without too strict deadlines and only as much backwards compatibility needed as you can provide, as well as buyin from whoever is paying you), then unironically go for it.

That's way fewer projects out there than people probably think. Plus, a lot of the wrongly started rewrites will go up in flames. Oh and it's probably also going to go wrong if people don't actually know the new language or tech.

It always goes wrong. I've done it like 3 times now. Migrated from one language to another, and then it was taking too long so we rushed it (and also no one was proficient in the new language), so it became an utter mess. So now (3 years later) we're spending another year un-effing it (or trying to). Another one we switched data backends which yielded no performance benefits nor new features and was also rushed. And the 3rd was mostly fine (because it was small) except that it was deprecated like 6 months later because of another mandatory migration. Millions of dollars just wasted writing and rewriting things with little to no benefit.

It’s a bad reorg if it doesn’t take into account on which service(s) which team is working.

Also a team sized service is IMHO is a lower bound, not upper bound. I’ve seen a small team working on tens of micro-services and IMHO it is over engineering (but there are exceptions when small micro-services make sense).

Ya, my team owns at least 4 services. It's a mess. One is deprecated, one calls the other 2 which just adds latency and redundant data fetches, and neither acts as proper 1st party API (there's always a debate about what service to call when another team wants to integrate with us), and the 4th is some pubsub thing which is invoked a completely different way, so that's one kinda OK. The most annoying part though is we still don't own the entire vertical. There's another 2 services downstream of us, and at least 1 upstream, so we're always at their mercy to accomplish anything.

In a redundant system, MTTR rapidly starts to be the dominating factor for high availability.

These guys seem to have the right approach.

Anyone else getting access denied on the page?

You must be in Russia? Just switch on VPN.

Ironically, I had to turn off VPN. I don't know why people put strict restrictions for static web pages.

Reply on news.ycombinator.com