Hacker News Reader

Comment by acroback | Hacker News Reader

This is a classic case of not understanding micro services and trying to fit a problem around a tool.

At work, we have close to ~50 services(no one calls them microservices), but they do not suffer from this brittleness. We segregate our services based on languages. So, all C services go under coco/ , all Java services go under jumanji/ , all go services go under goat/ , all JS services go under js/. This means, everytime you touch something under a repo, it affects everyone. You are forced to use existing code or improve it, or you risk breaking code for everyone else. What does this solve? This solves the fundamental problem a lot of leetcode/hackerrank monkeys miss, programming is a Social activity it is not a go into a cave and come out with a perfect solution in a month activity. More interaction among developers means Engineers are forced to account for trade offs. Software Engineering in its entirety is all about trade offs, unlike theoretical Comp Science.

Anyway, this helps because as Engineers we must respect and account for other Engineers decisions. This methods helps tremendously to do this. No one complains, everyone who wants 1000 more microservices usually turns out to be a code monkey entangled in new fad, or who doesn't want to work with other Engineers.

You want to use rust? There is a repo named fe2O3/, go on. Accountability and responsibility is on your shoulders now.

If you think about it, an Engineer is tied to his tools, why not segregate repos at language level instead of some arbitrary boundary no one knows about in a dynamic ecosystem?

View on HN

I'd wager that microservices, a lot of the time, are basically used as a management structure rather than for their benefits as pure tech, so less mature teams can silo themselves off and avoid communication (e.g. "I can work just on my backend image processing bit without dealing with the React guys now", "now the CTO won't be on my back so much," or whatever).

The irony being that anything approaching SOA (or microservices) requires exactly the same amount of communication. More likely they require more since it's almost certain that such a decision introduced chaos.

I'd wager that microservices, a lot of the time, are basically used as a management structure rather than for their benefits as pure tech

A lot of software is used to control or impose someone's will on others organizationally.

Very well said.

This is why things like ticketing systems and other workflow tools almost universally suck. Manager needs x field to run some report so x field becomes mandatory and the software is a pain to use because know one apart from that one manager knows what the field even means.

Currently our JIRA kanban board is crippled because someone just had to take a simple system and impose a workflow that can't be deviated from.

I agree, though I'm not so cynical about it. Well-defined API contracts are themselves a communication mechanism. If I provide an API, I am declaring that if you interact with me in a given way, I will behave in a certain way. Given that one of the hardest parts of scaling an organization is the boundaries between individuals and teams, providing a structured mechanism to define system behavior is incredibly valuable.

Well-written APIs are SLAs on steroids.

I hated that in our monolith Java web app, any random team could come along and pepper our team's module with global variable lookups and short term hacks that never get cleaned up. Even though we see the changes often times they are urgent changes needed right now, accompanied with promises to clean up later (which usually doesn't happen).

The nice thing about a SOA architecture is that it makes it harder to do this kind of cowboy programming. Yes maybe some quick business wins are harder than in a monolith, but I think (at least in our org) the cleaner architecture pays off by letting us move quicker on a different class of product projects since there is less technical debt.

I had enough of this stuff in my case so I locked repo down so only members of a team can merge PRs and no one can merge anything without a PR.

Isn't that a technical solution to an organizational problem?

Yes. And that's okay, sometimes. To me, the difference is whether it's an organizational challenge or an organizational dysfunction. There will always be challenges inherent to operating a business—and technical solutions often play a critical part. However, if the issue is really an organizational malady that should be addressed, technical solutions probably aren't the right approach.

I think what was missed, in the article, is that the fundamental problem was centered around a shared architecture of destinations and shared code.

You cannot possibly have every destination be a separate repo and then have the development lifecycle of your shared code be so active that it ultimately puts at risk the architecture of your entire organization.

What makes shared code so perfect is having stability such that you extract your variant code into your non-shared code. Shared code should evolve at a much slower pace than your non-shared code or you risk this very outcome.

Microservices are not dead, nor are they the solution to everything. We need better architects.

No we need science and theory to definitively define necessary architectures that are appropriate to certain use cases.

An architect is just a another man with an opinion.

We have these things. A better architect implies that one is trained in analyzing and producing these architectures using the science she/he was trained with: https://www.sei.cmu.edu/education-outreach/credentials/crede...

The certificate means nothing, the training and instruction is priceless. Add to that things like TOGAF and a deep understanding of the current state of existing architectures and you'll understand what I originally meant, but failed to explain.

UPDATE: misspelled "meant"

That's not science or proof. Thats a course on design. By science and proof I mean like definitive logic. a^2 + b^2 = c^2

The closest mathematical formal system I've seen that dictates architecture is category theory. But even this system doesn't say anything about being "better" or more efficient than another design pattern.

It's a bit like asking to replace medical doctors with science and theory.

We need science and theory, and skilled architects who can apply it to the problem at hand.

The practices of Doctors are based off of science and theory. Design patterns and microservices while technical are not based off of science. They are ideas without quantitiative basis or science.

Model checking is based off of mathematics and can be verified or, with some elbow grease, proven.

I'll take a TLA+ specification over a diagram any day.

What do you mean by model checking? Usually anything with the keyword "Design" like design patterns for microservices have no science or mathematics to back it up.

You may be technically correct, but I'd argue it's generally more useful what we do have: practical experience. When one says "we need better architects", to me that implies that our current architects are creating design patterns and reference implementations that are both 1) not practical, and 2) don't adhere to known-good best practices.

Systems design in real life is an "artistic science": there are always known limitations (and some expected unknown ones) that rule out the theoretic optimal design for good reasons. The problem is that limitations and compromises are often not disclosed, and that many programmers and architects are too inexperienced to really grok the implicit meaning behind specific design decisions.

So we struggle along, with bloggers, researchers and FOSS contributors halfheartedly collaborating to make point improvements as solutions are discovered. Stodgy enterprises suffer, big tech companies make decisions with global impact, startups are left wondering WTF to do, and the rest of us largely don't care. Why? Because nit picking doesn't solve business problems [almost ever. I'd argue this point in cases of things like SCADA systems and other mission critical control systems.].

The thing with practical experience is you can put two engineers with practical experience in a room and they can argue for days about a architecture or design pattern.

Nobody argues about which algorithm is better for sorted data sets: linear search or binary search. Theory already establishes one is faster than the other, but no theory establishes which architecture is better than the other.

A model checker is a software program you use to validate a given mathematical model of a system. If the proposed properties of the model hold the model checker will let you know. More interestingly if the properties fail to hold you'll get a trace back into where your assumptions fell apart.

TLA+ is one such system that includes a language for writing models and a model checker to verify them for you. In the context of microservices you would write a model of your services and the checker would help you to verify that certain properties of your model will hold for all possible executions of the model. Properties people seem to be interested in are consistency and transaction isolation. You can develop a model of your proposed microservices architecture and work out the errors in your design before you even write a lick of code.

Or if you already have a microservice system you could write a model of it and find if there are flaws in its design causing those annoying error reports.

Amazon wrote a paper about how they use it within the AWS team [0]. Highly worth the read. And if any of this sounds interesting I suggest checking out Hillel Wayne's course he's building [1].

[0] https://lamport.azurewebsites.net/tla/formal-methods-amazon.... [1] https://learntla.com/

Actually computing science is the most well understood science. Physicists can only give results with a few millionths* of accuracy, a computing scientist can prove a theorem about his subject matter with perfect accuracy.

Everyone else, including doctors, who call themselves scientists are just trying to float on the cachet physicists earned with their astonishingly good predictions. Properly speaking, they are phenomenologists . Please note that I'm not saying what they do isn't of great societal and intellectual value! The study and categorization of phenomena is certainly a noble enterprise. But none of them can make predictions good to 9 decimal places.

*might be billionths or quadrillionths by now in QED, but doesn't change my point.

You are diving to deep and talking about the difference between formal logic and empirical sciences. I am not talking about that.

Things like algorithmic complexity are well understood and formalized but design patterns are not a science nor has the concepts ever been formalized..

There is no theory or formalized system that says monolithic is better than micro or vice versa, it's all opinion. That's why its' called design.

Formal logic is a tool used by computing scientists and that they made almost incredible contributions to. It’s a tool that’s available to other scientists too, and the more rigorously minded ones use it.

For your latter point, it’s a matter of elegance, which is a pretty way of saying cognitively manageable. Think of epicycles vs Newtonian mechanics as an analogy. With enough epicycles you can compute the same result, but Newton’s approach is still a clear scientific advance.

Your point about design is well taken. Any given design is analogous to a theory. So we should aim for the simplest and most cognitively manageable design that satisfies our needs. That’s not literally formalized, but it’s a well established principle with an excellent record.

I agree that we need more empirical investigations of software 'diseases' and 'cures'.

That being said, practical medicine is much less scientific than many think. There is a lot of master/apprentice learning going on, just as in software engineering.

I don't even think we need empirical investigations. A formal logic system from axiomatic rules for architecture is really enough. I use the term science here incorrectly. I don't mean experimentation. I mean formalized logic. Like graph theory, complexity theory, number theory.... Architecture Theory.

In the world of math you don't need empirical data to verify a point. It's all logic derived from a small set of axioms.

Ah, here I disagree. Architecture is usually influenced not only by the software requirements, but also by many non-technical constraints (how is your development organization structured, how skilled are the devs, how sure are you that requirements will not change over time, etc.). There are just too many soft factors involved to 'compute' a solution.

> ... the fundamental problem was centered around a shared architecture of destinations and shared code.

From an architectural perspective, there is absolutely no difference between a micro service and a library. The only real difference is in the dispatch mechanism.

The problems around configuration management are the same problems we've had as programmers for decades. It's just that the people who are keen on micro services are usually not old enough to have experienced the pain in a different context.

Should shared libraries be in different repos, or should you put everything in a single repo? How do you deal with versioning? What happens if one app wants version 1 of the library and another app wants version 2? How do you deal with backwards compatibility of the API? Do you make a whole new library when you decide the old API is incompatible with the new vision? Blah, blah, blah, blah. None of this is a new problem.

I can't remember which version of Windows it was (maybe 7?) where they were seriously delayed mainly because they had so many programs using different versions of libraries. Integrating it at the end was apparently complete hell. Since they wanted to have separation of responsibility in their groups, each group was just pounding away implementing the features that they needed, but not integrating as they went -- because that would mean lots of cross team communication. The exact same thing is likely in a large organisation with a ton of micro services.

There isn't just one way to solve the problem. Mono repos and monoliths help in certain ways and cause problems in other ways. There are other techniques as well (should we implement ld.so for micro services? :-) ) But as you mention, the real answer is that the solution requires humans, not technology.

> From an architectural perspective, there is absolutely no difference between a micro service and a library.

I’ll stick to linking glibc, friend architect. Performance is also architecture.

Microservices do have various organizational tradeoffs. Individual teams can now own deployment, operation, language, and tooling choices for better or worse. This is probably advantageous for large companies where the number of teams scales beyond what a hierarchical control structure can support. In other words, a single devops team can't make good decisions and prescriptions about languages, tooling, deployment, etc nor effectively react to feedback from the dev teams. Control (power and responsibility) becomes distributed instead of centralized, with all the tradeoffs that entails.

"From an architectural perspective, there is absolutely no difference between a micro service and a library. The only real difference is in the dispatch mechanism."

I always tell people if you can't write and maintain a library then don't do microservices.

Yes the kids have just rediscovered the move from COM to DCOM.

It's so silly that we are wasting so much time rebuilding existing things, poorly.

COM wasn't a picnic to work with back when it was the new thing. The toolchain sucked. I'll never write another line of MIDL in my life.

Agreed. Although with tools like Visual Basic it was quite nice to use. Now, DCOM was another story. Pretty much everybody I knew who used it started to become suicidal after a while...

I still maintain a (third party) app that requires DCOM.

MDIL has been rejuvenated for C++/WinRT, the new C++17 based projection for UWP.

> "should we implement ld.so for micro services? :-)"

Working exactly along those lines this week on multiple services exposed over REST APIs, I was wondering if tools exist to check compatibility between them. Said differently,

- I have a `swagger.yaml` for my service A managing chipmunks and it says endpoint `/chipmunks` supports an 'color' query parameter.

- In service B, I have a `handleToServiceA` that encapsulates calling A. Then say I write `const chipmunks = handleToServiceA.getChipmunks({'colour': 'blue'})`.

Are there (whatever the ecosystem) tools that would read serviceA's swagger.yaml, detect my error in serviceB (color -> colour) and report the issue at compile time rather than run time?

I think this is one of the standard options: https://docs.pact.io

At work, to alleviate this issue, we use a client library for service A (i.e., one that provides service.getChipmunks(color='blue')). This gives you the compile time check, assuming the client library is a compiled language.

One step better is autogenerated client libraries where a new version is created every time a new version of swagger.yaml is deployed. However, I don't know an open source project that does this.

> Should shared libraries be in different repos, or should you put everything in a single repo? How do you deal with versioning? What happens if one app wants version 1 of the library and another app wants version 2? How do you deal with backwards compatibility of the API? Do you make a whole new library when you decide the old API is incompatible with the new vision?

Could you share methods that worked well for you? Asking because those question come up a lot and there never seems to be any conclusion.

You’re oversimplifying.

Architecturally speaking, there are some similarities between micro services and libraries because they’re both forms of modularization and usually have an API, but there are some stark differences beyond the “dispatch mechanism”.

The main difference is that a service’s deployment lifecycle is completely up to the service admin. Microservices are like websites - they can continuously evolve (within their API’s contract) without asking permission from consumers. This is their main superpower and why they’re a way of scaling a development organization without slowing it down too much.

In the case of a shared library, it’s completely up to the host admin as to when to upgrade. In the case of a static library, it’s up to the consuming software to determine when to upgrade. A service can upgrade when it feels like it.

Issues of API backwards compatibility, forwards compatibility, extensibility, self descriptiveness, versioning, etc. are old issues but usually have different answers when upgrades are truly happening all the time and not just in theory. It tends towards much fewer hard versions and more evolutionary backwards compatibility.

IOW, microservices aren’t a cure all, but they do encourage a set of behaviors. Many articles detracting from them seem to have not wanted those behaviors in their org in the first place.

You are assuming that microservices must be deployed on the internet, however they can be deployed on another type of network or on a single OS installation, which would prevent them from being completely transparently updated.

Shared libraries can be an implementation of micro services. On a platform (e.g. Android) it can be that a shared library is updated and then all consumers are forced to update.

No, I’m assuming they must be deployed on an IP network of some sort, not necessarily the Internet. A microservice by definition is a networked service. The whole REASON the term was coined was to differentiate from services which often were developed and deployed monolithically, as opposed to truly autonomous runtime processes (aka. bounded contexts).

Services, whether SOA, web services, messaging services, or network services etc, as in SOA, describe a client/server architecture with an API. Usually these APIs are designed by the principles of Domain Driven Design, where different teams map to bounded contexts that have their own published API.

Microservices are a form of SOA where each API also runs in its own process and thus has independent deployment lifecycle. Many in the SOA world advocated for this 10-15 years ago (and often ignored), and now the industry has gone back and coined a term for this practice.

In an organization that does microservices properly at scale, like Amazon, you have teams that build and run and upgrade their service autonomously from others. Read the Steve Yegge rant about Amazon and Platforms to understand this. It allows tremendous parallelisation of effort and allows for thousands of deploys to production daily without breakage. This is hard to pull off though, and a new initiative doesn’t generally won’t more than a small handful of services.

If it’s a shared library, then call it a shared library. It has a completely different lifecycle from a microservice. In your Android example, shared libraries by definition are controlled by whomever can dictate OS updates, not the app developer.

Yes, couldn't agree more. The O in SOLID should be adhered to in this case. Open for extension, closed for modification

I would flip this argument around 180 degrees. Once you have any system that's built by large numbers of people, you have to discuss and negotiate your interfaces. You can either do this implicitly, by having a shared codebase and making sure that your test coverage and code reviews guarantee backwards compatibility, or explicitly, by defining service interfaces and black-boxing behind them.

Yeah. We're currently in a second push to introduce microservices and this time I'm on board. The first time was more about enforcing code ownership and such, and that was going to be a mess.

Now, people are pondering things like: Ok, management of elasticsearch has performance issues and it's a general pain if the elasticsearch documents change. So let's try to move the schema of elasticsearch documents into a strictly semver'd artifact. And let's move management of our search indexes into a service depending on that artifact, so the schema changes in a controlled way. And ops and the search team can scale and optimize the service as they need to minimize search outages.

Creating smaller services based on problems is a good thing. Creating smaller services because of ... reasons... not so much.

I'd wager that microservices, a lot of the time, are basically used as a management structure

Conway’s Law in action, or in reverse https://en.wikipedia.org/wiki/Conway's_law

What is the value proposition of tieing an ingester, parser, transactional service (e.g. all written with go), and then tieing together your api, client app, mobile app (all in js), just because they share a common language?

arguably they aren't close to each other in tiers of the stack, and don't really overlap. The sorts of libraries you might choose to use to do something could differ, (e.g. json processing - in the front end, you'd pick usability and security over a lower level more optimized transcoder for the api).

That said, I agree with your core premise- not understanding, and chasing a nail with a hammer, but geekily named repos per language just sounds like someone who doesn't understand how something git works...

This is a classic case of not understanding micro services and trying to fit a problem around a tool.

This sentence needs to be repeated for everything on the 'wrong end' of the hype curve.

This is a classic case of not understanding NoSQL and trying to fit a problem around a tool.

This is a classic case of not understanding OOP and trying to fit a problem around a tool.

This is a classic case of not understanding dynamic typing and trying to fit a problem around a tool.

So what's the 'respectable end' of the veneration curve? Does it have an inverse sentence?

  This is a classic case of generally understanding 
  Turing completeness and eventually, inadvertently
  implementing a half of Common Lisp in C anyway.

Something like that?

Our industry is too immature to have veneration for things that are old, I think.

> This means, everytime you touch something under a repo, it affects everyone.

This is horrible at scale.

> This is a classic case of not understanding micro services and trying to fit a problem around a tool.

That much I agree with. TFA even acknowledges that, in the conclusion. Not in so many words, but they basically admit they did it wrong.

>> This means, everytime you touch something under a repo, it affects everyone.

>This is horrible at scale.

I just want to reiterate this. In the early 2000s I worked in the online platform group at EA. The list of things done poorly there was long, but picture:

* 40+ engineers

* Monorepo with hundreds of thousands of classes; all code deployed to all servers.

* Hundreds of different services running across thousands of servers.

* Communication based on Java serialization, so all code had to be deployed to all servers at the same time.

* Deployments (and thus downtime) sometimes lasted up to a hour. Worldwide audience; it was always in the middle of someone's day.

* Rational Clearcase for version control. It took nearly an hour to sync to tip.

Pretty much every morning you'd come in, spend an hour syncing, find that someone broke the build, hunt them down, and resync for another hour. Generally speaking the first few hours of every day were wasted for 40 engineers.

This was a very poor platform.

Sometimes I wonder how it's going there these days.

The Java serialization downtime was sad, but I died when you said clear case.

When they rolled out clear case in my team at IBM, I quit a few weeks later.

Funny that it was this bad at an actual tech company. What you described is pretty common in normal big enterprise, though.

> This is horrible at scale.

This can work fine at scale, Google does it with however many tens of thousands of engineers they have these days. Having everything in a monorepo doesn't solve the communication problem, it doesn't prevent solving it either.

Sort of. Google is also well known for having a dozen chat apps. What most people don't know is that there are also likely a dozen different libraries/utilities/interfaces performing near identical services just because an engineer or product team weren't 100% satisfied with what already existed. So yes, it's a monorep, but holy heck there's a lot of duplicative cruft.

> all code deployed to all servers This is the main point of the post. Google most certainly doesn't do this, even if they have all their code (or at least, all their private code) in one repo (piper).

> all JS services go under js/

Aww, no clever name or the JS services?

badlanguage/ is too long

goodbits/ is smaller though ;)

Such a tired view/joke. Python is just as bad but doesn't receive nearly as much hate

JavaScript is a great language because you can run it on pretty much any machine.

Buts it's especially terrible because it was designed by a guy without much language expertise in a handful of weeks.

> Buts it's especially terrible because it was designed by a guy without much language expertise in a handful of weeks.

The core isn't too bad; I'd argue that the main reason why it's hard to work with is because you don't have tight control over your execution environment, mostly since the language wasn't really standardized until pretty late. It also solves a much different problem than most languages since it has to optimize for better UX (not crashing the whole program on exceptions, for example).

I'd like to hear why you think Python is as bad

Horrible package management story. Cross platform is a headache. Slow.

Cross-platform in Python is not a headache?

Yeah, it's a bit slow. It's faster than Javascript, though. The best benefit is flexible number size, so it's good for some types of computation.

What improvements in package management would you want? At least it has some form of management, unlike C/C++...

2 or 3? Or all of the problems that come with choosing between the two (no, it is still not an easy choice).

Mypy coverage, even in the stdlib, is awful. When it comes to thirdparty, mostly nonexistent. Mypy feels so young - I love the team, love their work, but I still run into cases where inference fails when it shouldn't, where error messages are extremely unhelpful, etc.

Slow.

Plenty of 'wat' like mistakes: https://stackoverflow.com/questions/3270680/how-does-python-...

Exceptions everywhere, for control flow even - iterators are implemented with exceptions.

Absolutely AOT unoptimizable. Pypy's cool, never got it working for my use case. In theory a JIT could help.

Speaking of calling out to C... you think you're writing in a memory safe language, but actually, you're writing in a memory safe language that's probably been hollowed out and replaced with a fast C implementation. But it's actually worse

https://hackernoon.com/python-sandbox-escape-via-a-memory-co...

Exploiting C code loaded by Python is like exploiting C code from the 1990s.

No parallelism. Multiprocessing? Good luck with that - pay the cost of pickling, pay the cost of an additional interpreter, pay the cost of debugging hell.

An ecosystem split in two, and don't let anyone tell you otherwise - a few hundred top packages moving over after many, many years, is a sad state for a language that was known for having an absurdly large ecosystem.

I could really just go on and on and on, but at some point it just feels mean.

Huge respect for the project and the team but Python has made mistakes (as all languages do). They were understandable mistakes, but they were mistakes. It's fine for some things, but there's plenty wrong with it, just like there's a ton wrong with javascript. But javascript gets probably 1000x the flack.

> no, it is still not an easy choice

Yeah it is, choose 3. Python 2 is reaching EOL.

> Mypy coverage, even in the stdlib, is awful. When it comes to thirdparty, mostly nonexistent. Mypy feels so young - I love the team, love their work, but I still run into cases where inference fails when it shouldn't, where error messages are extremely helpful, etc.

Yeah, the typing stuff is a pretty big disappointment. The ergonomics are pretty terrible (probably because they wanted to push as far as they could without introducing more syntax support for typing). Mypy isn't just young, but it's buggy and its codebase was a sloppy mess last I checked. There's no support for recursive types (you can't define a JSON type, for example). And it absolutely falls over in the face of common libraries, like SQLAlchemy, which are too dynamic for it.

> Absolutely AOT unoptimizable. Pypy's cool, never got it working for my use case. In theory a JIT could help.

Yeah, Pypy is the best hope for Python's performance. They're making great progress, but I also couldn't get it working in our Python 3 codebase (Numpy and Pandas installation issues).

> No parallelism. Multiprocessing? Good luck with that - pay the cost of pickling, pay the cost of an additional interpreter, pay the cost of debugging hell.

Yeah, this is a real pain point. Some die-hard Python folks say otherwise, but there's really no good parallelism option for lots of workloads. Pickling is just too expensive.

> An ecosystem split in two, and don't let anyone tell you otherwise - a few hundred top packages moving over after many, many years, is a sad state for a language that was known for having an absurdly large ecosystem.

This hasn't been a problem for me for years. Most things that have seen active development in the last 5 years have good Python 3 support. The only time I've run into a Python-2 only utility, it was 7 years stale. Well, except for Centos's `yum`.

These are all reasons I like Go, by the way. Super fast, great tooling, and fairly stable (except for the package management story). I do wish there was a lightweight scripting language with a great VM and real parallelism--something like JS without the inheritance, OO baggage, etc; just objects and arrays and functions running on a JIT VM like V8 but with parallelism as a first-class citizen a la BEAM. And preferably optionally typed from the start, in a way that the runtime could leverage for optimization purposes.

> Yeah it is, choose 3. Python 2 is reaching EOL.

Easy to say. Tell that to Google and Dropbox - Guido has worked for both and they're on 2. Tell that to the companies that can't afford the creator of the language.

3 is not the easy choice.

Anyways, maybe try nim? I've heard good things.

"2 or 3" wasn't a choice for Dropbox/Google, and even if it had been, it's not a choice that concerns you. All the stuff they produce publicly is Python 3-compatible.

My point is that tons and tons of companies, with many billions of lines of code, do not have an easy option.

Plenty of the environment still has not moved over even outside of companies.

If you're building something new as the original phrasing implies, choosing 3 is easy. If you need to integrate with 2, then you're in a tougher spot.

Re your last paragraph, you're describing Lua.

I don't know the state of its coroutines though as I've never used them.

I should have mentioned that "async coroutines" (i.e., goroutines) are also a requirement. Also, I understand that Lua doesn't have much of a standard library and isn't really used as a general purpose programming language (compared to Python, for example)?

Your understanding is correct, it's an embedded-first language. It also makes it very hackable which means that I fully believe golang-like goroutines would be achievable in Lua.

People hack the `continue` statement into it all the time, it wouldn't be terribly surprising.

I'm a Python guy, but there are lots of things about JS that make me jealous. For example, the main implementations are quite fast, and the tooling is good. In Python, there is Pypy which doesn't get enough investment, or CPython where your only optimization lever is "rewrite it in C!". TypeScript also seems quite a lot nicer/less-broken than Python's mypy. Lastly, I like that JS has actual, multi-expression lambdas and a pleasant syntax.

Some things are more pleasant in Python--everything is sync by default, there's less churn, the standard library gets me a lot farther, it's less permissive (no 'undefined is not a function' nor '!= vs !=='), etc. Both languages serve similar niches well, but the feeling I get is that JS is quite a bit faster and nicer for IO-heavy workloads (JS just has a more mature async story than Python) while Python is generally more intuitive and perhaps better for general application development. But for most things, one isn't dramatically better than the other, and they're both a good deal more disappointing than Go. :p

Ok so if I want to build a fledgling project all I need to do is leverage a language nobody uses yet. Lolcats ftw.

Seriously though constraining dev's to use a monolithic repository in the name of discouraging cowboys is like tying people's legs together to ensure they walk in an aligned direction. Sure it works but seriously unnecessary pain. Your company should invest in integration tests

what happens when the java code breaks the c code? Or does everyone roll-their-own everything in each language?

This means either your contract agreement between services is not language agonistic or someone did not do their job when Interface schema was changed during reviews.

Just like any other software, interfaces live in separate repos e.g protoplasm/ for proto-buf definitions. avrobber/ for avro and so on so forth.

So, any change to protoplasm/ triggers automated tests on all other services irrespective of boundaries.

ok. But _much_ easier to have compile time checking between projects in the same language/repo. Not saying this approach is wrong, but it's a little like riding a unicycle when a perfectly good bicycle is sitting right there.

It's much easier to drive a car than an 18 wheel truck too, but the latter have their uses.

I think I did not present this correctly, I apologize for that.

What I mean this this. Any IDL interfaces live at their functional boundaries. e.g all proto definitions shared internally by java services live under jumanji/proto/, all proto definitions shared internally bu Go service live under goat/proto/ and so on so forth.

But anything which is shared in a public manner e.g any proto definitions between Java and C live outside either of these repos in top level common_proto/ repo. The build system is configured to take care of pulling the correct proto version from common_proto repo.

We use maven for this, but nothing much we can do about that. Build systems are ugly and we have to live with maven in near future.

EDIT: Oh that said, stop using required field in protobufs if you are stuck using proto2 like us, whoever decided to add required field to proto2 grammar made a mess. Things change and deprecate repeatedly, gazillions of required fields are annoying and recipe for disaster.

Sharing schemas and contracts in a distributed architecture is a hard problem. The interesting thing is these are central to communication, so we should definitely talk as a team about them - if anything, that's probably the one thing if you had to choose what to talk about. Then the INTPs can scurry back to their holes and continue creating stuff while the ENFPs form a committee for the next pub walk.

Full disclosure: INTP here.

Sharing schemas and contract's isnt a hard problem. Toss them in a shared repo and push it to artifactory. Anyone who updates it increments the repo. Done.

Are you sure that this dichotomy is a good way to approach the situation?

It just seems like it just creates an us vs. them ideology.

The grandiose tone of your message suggests you believe you are giving some insteful advice but all I see is a single suggestion grouped with a lot of that weird marketingspeak every single programmer, for whatever reason, feels they need to emulate on their blogs.

Amen to that. Just by curiosity, what is the barrier of entry for adding a new language in your ecosystem?

Totally technical considerations which engineers must present and justify.

e.g When we added Go, we had to present what it brings? A good testing framework, light weight(subjective), goroutines(which fit our use case), in built benchmark support, mature support of third party packages etc etc.

Aren't literally all of the above present in Java?

No, we do not want to pay extra to run Java services bloating our measly VMs. Also, we found that it is far easier to get consistent latency with Go code than with Java code.

Again, YMMV benchmark for your use case.

That said, Java doesn't have go routines, doesn't provide low level access to memory, benchmarking framework is easy to work with in Go. Go gives access to sync Pool which when use correctly gives extremely good gains.

One of our measly dual core VMs, can do more than 40k QPS with predictable latency at 99 percentile. With JAVA oom panics were common and latency percentile was all over the place. Sure, you can perhaps tweak the Java code, but it was far easier to do with Go.

They are. Add around 1.5 gig of necessary heap per feature you use.

Like the others said, our JVM apps would require at least 1GB for the JVM and then 1-4GB for heap, depending on the service. This made the average JVM service require about 4GB of RAM in the container scheduler.

Meanwhile our Go services almost never had a working set larger than 256MB and most didn't even need that. We could schedule on average 16 Go services for each JVM service.

I love the low overhead.

Out of curiosity, was it replacing an existing language or developing new functionality?

New functionality, breaking existing perfectly fine and working functionality is a big no.

One of our core services is written in C, it never crashes, it sustains QPS no other service can in the entire ecosystem, why change it to something new unless there is a solid reasoning to it?

I kid you not, one of the companies we integrated with gave us 2 64 Core VMs in their private cloud to run our service. We run 3 instances of our process on each VM along with Redis taking about 60GB of memory in total, with 59.996 MB of memory taken by Redis. And it was written in Go. Apparently, we miscommunicated that we will be using a GC language for this service. They assumed Java and based on their experience allocated 2 64 core VMs with 256GB memory each :D.

why js/ didn't get a cool nickname?

Can you be the Jack Doneghy to my Liz Lemon?

Please?

I wonder why those fancy names for all except JS? I would have called it jizz/

Haha, I made that up to gain free internet points ;)

Reply on news.ycombinator.com