Hacker News Reader

The problem is that there are so many developers now who have never had any experience of anything that isn't some botched attempt at microservices. The idea that it's possible to encapsulate code and separate concerns in any other way is foreign to them, and an "API" to them is 100% synonymous with a rest/grpc interface. So there's nothing for them to revert to and they are doomed to repeat this pattern, clearly with the impression that this is what app development is.

Meanwhile a lot of the industry is trying to tell them that their problem is they haven't separated things enough and should be using lambdas for everything.

> So there's nothing for them to revert to and they are doomed to repeat this pattern, clearly with the impression that this is what app development is

The only practical solution I have found for this: Carefully select a Padawan and guide them gently away from the forces of evil until they have gained enough situational awareness to spot these patterns and defend themselves. Not everyone can be saved, and at this point I fear that it is a majority.

If someone goes too far into cloud insanity, there is (in my experience) very little you can do to bring them back down to reality. At least, not on time frames the business owners seemed interested in when looking at new hires. I have had a much easier time taking in someone totally green and getting them happy/productive on monolithic software than I have with uService/AWS-certified 'experts' (et. al.).

I tried that for a while but now my reviews are basically:

"No, stop, don't."

https://youtu.be/uVdDXeYM4ss

It's like I'm talking to children, they think I'm just old and stuck in my ways and that I just don't understand.

Eventually my prediction comes true but none of them have ever said anything, they just go implement what I suggested and act like they solved the world's greatest mystery.

I guess it's easy to forget my comments from months earlier, it's almost as if I've seen their code before and know how it ends.

Ah yes, the curse of Cassandra.

https://en.wikipedia.org/wiki/Cassandra_(metaphor)

This is happening to me, its driving me nuts. Any advice on how to get out of it?

One thing that has periodically worked for me is the application of some fun infographic-tier latency figures to really drive home the argument for why distributed anything generally sucks ass. I.e. Would you rather that customer transaction either be:

1) A direct method invocation resolved within the same L1 cache contents.

or

2) One quick network hop just 5 milliseconds away?

Assuming worst case processing semantics (global total order), you would be able to process ~10 million times more customer transactions per unit time with option 1 vs option 2. This is seven orders of magnitude.

In my experience, not a whole lot is truly worst case, but most complicated & important business systems (banking/finance/inventory/logistics/crm/etc.) are pretty close if you don't want to be chasing temporal rabbits around all day.

I worked at a place that had a microservice for validations. You made a request with the item (singular) to be validated and the rule to validate it by. The most common validations were things like "is this value greater than 15" or "are these values equal".

Nobody else saw the problem here. Establishing a http connection, serializing a JSON document, deseralizing it on the other side, then doing "14 == 15" anyway before going back the way it came. Lighting up a hundred thousand lines of code vs one.

This was done as it iterated over large files, millions of items, millions of requests.

They were genuinely confused why it was slow.

Your story both makes me want to laugh and cry because I've seen this pattern so many times.

These days when I do architecture interviews, I always make sure the problem can be solved distributed or monolithic and when they do distributed, I asked them why and to explain the trade-offs. If they do monolithic, I ask them why not distributed. Etc. For senior engineers, it's such a good indicator of whether or not they've truly worked in distributed systems and understand the human side of the problem.

The big O stuff is great, but the human and organizational costs are easily as important to assess as the computational and memory costs.

I am sure "ValidationService" made perfect sense to everyone the first day it was dreamed up. IIRC we had one named that too for a while back when we were lost in the dark forest.

This is actually why I think we need more cloud and not less. The problem is poorly designed code, not monolith or microservice.

For example, ValidationService as described above could be replaced with OpenPolicyAgent and would scale much better.

That said, I could see someone pointing at OpenPolicyAgent and asking why they didn’t separate distributed updates from the policy spec as two distinct projects, this way you could re-use parts of OPA to distribute config files or feature flags. You can now too, but it requires the extra hop of compiling a function to answer.

Come to think of it, distributed state updating is roughly the same general problem that Kubernetes Operator pattern solves.

Another way of putting it, we have cloud primitives but not enough of them, maybe - or they aren’t well enough explained - to make the choices or restrictions of different architectures a bit more obvious? I especially look forward to a future when we get “design systems” for the programming we do with business logic in the cloud.

I’m over-emphasizing cloud here. I mean code you don’t directly have to maintain and write regardless of where it runs or who wrote it.

Scaling is not the issue

Perhaps I didn’t explain myself fully. OpenPolicyAgent distributes the policy so you can integrate[1] it with systems that need to ask for validation such that lookups are constant time and performed on the same machine but distributed - scaled - using code and data shipped to every consumer/integrator of the validation service, instead of making a separate call to validate. So you can have centralized control of your validation logic and data but have it run as part of a decentralized application or on the same host as part of a pod. That is what I meant by a design that “scales”.

That said, depending on the type of validation performed, zero trust systems for example are often built using centralized validation endpoints, so it’s not entirely a bad practice.

1: https://www.openpolicyagent.org/docs/latest/integration/

Reading your comments I get the sense that you consider system architecture to be reducible to 2 possible choices (monolith and m.s.) and you are confusing distributed systems with micro-service architecture. Would you consider a classic 3-tier (api - domain logic - database) to be a “distributed” system? Are micro-service systems “distributed”? (The answer is no to both, btw).

Micro-services were promoted for a variety of reasons. One of which was facilitating rolling out projects by consulting companies given the economic incentives for such businesses. Related was the VC driven development model that required rolling out features at greater speed than that of thoughtful development. Another was (remains?) the prevalent lack of competence in designing schemas (required for the basic n-tier system culminating in the database tier).

My advice would be to carefully construct an explanations of the scenarios when microservices do apply, and then explain why you are not currently in that situation. the best microservice examples I've ever found were for the addition of features to legacy systems, the ability to write minor additional things in alternate languages, and most critically, when the original code base couldn't be changed or was lost, and yet more functionality was required. There are valid reasons to add a microservice there are probably not good reasons to take a normal code base made by a modern company and completely shift to only microservices.

>My advice would be to carefully construct an explanations of the scenarios when microservices do apply, and then explain why you are not currently in that situation. I tried that once. Didn't work. End result was just pure pain.

Couldn't agree more - just like NoSQL vs regular old SQL - don't assume you need a NoSQL solution, until you actually prove you need it. probably 90-97% of solutions will be better off with a relational database; boring yes, but they 'just work'. Choose NoSQL only when you need them, choose micro services the same way.

If you 'think' you need NoSQL or a MicroService architecture - chances are you don't.

I don’t think I need NoSQL. It’s just that I hate having to map my structured data into tables and rows and joins and schemas and primary keys. A document|object|key-value store just usually maps better to the data I’m trying to model than a trumped up spreadsheet that requires me to speak a foreign language to it. Other’s mileage may vary of course. Just been my experience in the domains I’ve worked in.

(Ironically, I agree very much with your general sentiment, just not the example of illustration you chose)

You can do those in most SQL databases too, with the option of structured data if some of your data fits it

> having to map my structured data into tables and rows and joins and schemas and primary keys

You don't have to. You can just use JSONB.

I wish I could justify using SQL more. It's just way too easy to use Dynamo. Unless my data is complicated enough to warrant a relational model, it's going in Dynamo.

I don't disagree, it is so easy to build solutions on tp of dynamodb - and then I regret it when the access patterns change and the solution falls apart -thats when I wish I had stuck with tried-and-true relational.

That is kind of the problem, DynamoDB - and other NoSQL database - allow you to just start building your app, without any real thought about what data you will need and how it will be used - but you are often shooting yourself in the foot to not map this stuff properly out in advance.

> don't assume you need a NoSQL solution

Likewise, people who only know relational DBs shouldn't assume they need an SQL solution. It's remarkably rare that you encounter engineers with real thoughtful experience with both and they have the experience to know which tool to pick up for different problems.

I think one mistake is using the term “monolithic” in comparison to “microservices”. To a whole new generation of programmers (and a fair proportion of older ones also) it has a negative meaning somewhere approaching “a huge and growing mess of smelly unstructured, non-modular code”. It is a negative term invented to compare against the glory that be “microservices”.

I prefer to use the term “Local Modular Application” in lieu of anything better?

To some extent I think "monolith" is actually changing connotation as people realize that microservices are not a silver bullet. For example, Shopify calls their system a "modular monolith":

https://shopify.engineering/deconstructing-monolith-designin...

What about just "service" or "web service"?

Over the past 10 years dev tools (eg. git, IDEs, laptops) have also become more performant by some small constant factor, 2-3x, which isn't massive in comparison to Moore's law stuff, but means that a larger fraction of new software projects can start monolithic and stay monolithic for longer.

Unless you're handling millions of lines of code and 1000s of developers, you won't need to abandon a monorepo or commit to expensive custom dev-platform investments.

> Unless you're handling millions of lines of code

When including our vendor's WSDL generated reference sources, we are well in excess of 1mm lines of code. I think even omitting that we are very close.

I cannot imagine us ever getting too big in those terms. Our full checkout is ~400 megs right now. If some day github says we are too big and need to break shit up, Ill move us to perforce or something even more ridiculous.

Our main product does a full re-build in ~45 seconds on a powerful developer machine. Incrementals for more likely work areas are closer to 8 seconds. Full monorepo checkbuild takes about 5 minutes. For reference, we do use .NET so JIT is an ally here.

> full re-build in ~45 seconds

That's cute. How about the test suites?

Likely faster, less flaky, less complicated setup and higher coverage than the integration test suite of a set of microservices.

Using Go stack here. Can confirm it can be done. We're at 8s to build and another 15s to run full test suite.

If a repo needs to be broken up for performance or organisational reasons, is there really a reason to prefer to integrate the parts as microservices rather than as libraries?

The main benefit of microservices there is that they can be independently deployed. If you have libs into a monolith you have to redeploy the entire thing, which may involve coordination.

I still mostly prefer the libs + monolith approach though.

10 years ago SSDs weren't common, and they are 10x performance improvement alone.

I go back and forth on this. Having a single service is definitely much easier. However, consider the following scenario: you want an application that asynchronously ingests data from different sources (whether they are files, streams, database stores, etc.), perform some computations and aggregations, store them to a local DB, then allow the user to display the result. Would it make sense to separate out the ingestion code into its own service, that calls out to another service that is responsible for storing to the local DB? The rationale is that writing to the DB is now controlled through one service (almost like a queue, where connections could be controlled), and the ingestion could be scaled up/down independently of the DB write service. Thoughts?

Your problem was using JSON instead of a typed language like Protocol Buffers or even Java RPCs, and that you were using separate repos for everything.

Microservices don't even have to run in separate binaries, let along separate machines.

> Microservices don't even have to run in separate binaries, let along separate machines.

If you're not running separate binaries, what you have is a monolith with well-defined module boundaries, not a set of microservices.

Which is actually much better than a distributed monolith (random microservices thrown together just because it ticks some boxes on managements resumes)

Agreed! I think a well-structured monolith is what most projects call for.

What I don't want is for people to buy the microservices hype so thoroughly that they start thinking "microservices" just means "well-architected system".

I guess a binary is pretty vague. They could mean they deploy a single container image or something.

If you have a single deployment unit, you have a monolith, regardless of how that deployment unit is structured on the inside. There's a lot of fuzziness about what "microservices" means, but some degree of independent deployability is pretty non-negotiable.

You can have of course have a single binary deployed with different versions for different services.

More commonly the binaries would differ but share a lot of code in a library of the monorepo.

I was fortunate to have a similar experience early on. That being said, I still firmly believe there are cases where "microservices" are appropriate, such as when you married to a specialized framework that only supports Python 3.5 but you want to use modern tooling for everything else as much as possible :)

I think this is a fair point. If your reasoning for developing multiple services is to shim legacy services, that is perfectly acceptable in my view. It's when you intend to start from zero with these things that the problems begin.

I would say if you begin a journey to shim legacy, it should also be a journey to entirely deprecate it. You can gradually consume an old monolith with a new one over time if you are deliberate enough.

Yes obviously there are use cases where microservices is a better architecture, but the point is it's very rare and most teams/companies never need it, because it would complicate their life unnecessarily.