Hacker News Reader

Comment by klabb3 | Hacker News Reader

This makes sense but I can’t really unsee the learnings from subject based addressing in messaging systems like NATS. It would make this article like one paragraph long.

The very idea that services should talk directly to another is flawed even on happy days without any migrations and such. There’s service discovery, addressing, load balancing, firewall configurations, rate limiting that really should all be part of one logical connectivity plane. In NATS, your services can run behind a residential NAT if you wanted to, all you need is to connect to the cluster and choose a subject space.

Reverse proxies is probably the most common modern solution though in things like k8s. I’m not wise enough to say if they are better from an architectural pov.

View on HN

Is this the NATS you're referring to here? Forgive the noob question, I just haven't come across that acronym before (except in the Network Address Translation context.) https://nats.io/

Correct. The messaging system. Not to be confused with Network Address Translation (NAT). Since I used both terms, I should have made that clear.

Thank you for clarifying, no problem — always fun to learn something new.

Architecturally speaking I've always liked building process-group middleware atop multicast, since it has the clearest mechanical sympathy down to the wire and can be built up to handle things like virtual synchrony or tuplespaces with potential for zero-copy I/O if you're careful. My favourite example of off-the-shelf implementations would be the Spread Toolkit¹𝄒², a distant second would be tib/rendezvous².

Unfortunately deploying multicast in practice requires having several infrastructure ducks in a row that remain hard to achieve in enterprise environments let alone public clouds, so the devops mafia generally disregard it and deploy sidecar proxies in a unicast mesh instead, with log-structured event stores as the pub/sub core. There has been some continuing research into multicast as middleware³ but to my chagrin (albeit not my surprise) the barriers of infrastructure mean it hasn't developed as an application layer nearly as much as the unicast approaches.

[1] http://www.spread.org but sadly now either defunct or classified, not sure which, the whole project kinda went suddenly silent ca.2018

[2] not zero-copy; you'd have to roll your own for that

[3] e.g. out of the CCRG at UCSC https://ccrg.ucsc.edu/about-our-work/research-areas/routing-...

Never used multicast outside of local networks, but I was wondering: does you like it for the perf benefits or are there practical advantages that makes multicast systems easier to reason about or program for?

I like the efficiency, but moreover I like low latency and highly observable solutions, perhaps because my first and most formative job was building hard-real-time interactive devices and this left me with a lifelong distaste for any system that responds more slowly than 100ms. Proxies inevitably introduce the latency of buffering, and this tends to go exponential to the point of timeout beyond 99%ile. Not saying network protocols don’t suffer at their limit too, but it’s more graceful and more directly instrumented.

Neither are particularly straightforward to reason about, but ordering guarantees (in the Lamport sense) are (in my experience) harder to ensure at regional scale with proxies and service meshes. There’s a counterexample, of course: at intercontinental and interplanetary scale we lack the ability to stabilise tree topologies with predictable forwarding latency and are forced to use service proxies and bounded timestamps on replicated data to achieve ordering guarantees instead, of which the best known example is possibly TrueTime. This trades interactivity for scale.

Multicast is definitely not easier to program for, and although I could attribute that to a shortage of high-level libraries, in practice the shortfall is mediated by the infrastructure dependency. This places it beyond the radar of most system designers in commercial environments, thus completing the elements of a downward popularity spiral. It’s a shame, but that’s tech.

There’s a software called Skupper[0] based on AMQP which does something similar. Instead of using legacy technologies like DNS and IP the applications using skupper connect based on application name and the routing to a service listening on that name is handled at layer 7.

Something like that seems like the way forward but I haven’t seen an implementation yet that truly frees devs from worrying about DNS names and IP. K8s can paper over them but they’re still under there waiting to bite you.

0: https://skupper.io/index.html

thanks, you made me realize I know nothing about anything

Im not a backend guy. I assure you that I’m not a magician. As I never had the fortune of being blessed with the knowledge of modern backend stacks like k8s. All I know is at a surface level. This probably makes me naive. At the same time, I see these problems people deal with and wonder why make it so difficult.

Nats is a at its core a pub-sub system. Everyone (clients and services) connect to the Nats server (or cluster but same principle). Then you can either publish or subscribe to messages, whose payload is typically simple json.

Anyway, the point here is that with such a setup, you don't need to worry about networking between services. How do I stand up new service, let’s call it Galactus? You connect to the nats server, then you listen for messages on a subject space like “galactus.*”. Now anyone else can talk to Galactus, without proxies, api gateways - Galactus doesn’t even need a URL, only a subject (this is what subject based addressing means). As a result, you only worry about networking once.

These systems have been around and greatly improved for all kinds of use cases, yet it seems like nobody is paying attention.

I don't think nobody is paying attention. Message oriented middleware has been around for decades and pretty much every company with a service oriented architecture is using it somewhere, whether as ActiveMQ, RabbitMQ, Kafka, SNS/SQS or whatever else.

The issue is that moving from synchronous API calls to asynchronous message passing introduces new complexity. It's not suitable for every problem. Most places I worked use a combination of synchronous APIs and asynchronous messaging to share information between services, depending on requirements.

Of course in some cases you can just share access to the same database or filesystem too.

> The issue is that moving from synchronous API calls to asynchronous message passing introduces new complexity. It's not suitable for every problem.

This has been a problem I've been trying to solve for years! Surprisingly, it's a fun problem space, mostly because I disagree that it isn't suitable for every problem.

The solution I've been converging on is quite interesting, essentially a non-linear programming language that you write linearly (similar to how CPU branch prediction runs code that hasn't executed yet ... just on a more abstract level). It's pretty neat in how it works under the hood, but on the programmer level, it feels like just another programming language; but like erlang/elixir, the execution can be spread all over the world, or just your machine.

Probably one of the coolest things was when I got optimistic mirroring working, where local code will "predict" what distant code will do and then allow your local code to run to completion but not commit the result until the remote code agrees with the prediction -- and if it disagrees the code has to rerun that portion.

Fun stuff, but still probably a few years away from a HN post about it... it's just a fun side-research project atm.

Sounds very interesting.

> Fun stuff, but still probably a few years away from a HN post about it...

Remember that partial results are very useful too. Personally I like reading posts that have unfinished ideas, with commentary. Eg “I needed something to achieve X and I settled on Y. That causes some annoying side effect Z, so I’ll keep looking for a better solution”. That stuff is even more engaging than “look at this almighty finished and perfect system”.

Heh. Most progress is done in error messages at the moment: “if I get this error, then it didn’t work, but if I get this error it did”. If I solve distributed systems unit testing, that will be nice and worth a post all by itself, but not quite there, yet.

> The issue is that moving from synchronous API calls to asynchronous message passing introduces new complexity. It's not suitable for every problem.

Well yes but request-response is simply a pattern that can be implemented on top of pub-sub extremely easily. Nats implements it with a temporary generated inbox subject for responses, so dead simple. This keeps application logic conceptually identical to traditional rpcs.

The gain though doesn’t come from the messaging patterns themselves though, but the addressing. Going from “I wanna talk to X” to “I wanna talk about Y” is an extremely powerful shift of mindset.

thanks, you made me realize I'm not alone

agreed; you made me realize something i'd not connected before: NATS has a bit of similarity to Linda Tuplespaces and I think Javaspaces, when i think about its addressing.

If everything goes through a single intermediary, isn't it awfully inefficient? Lots of duplicated bits on the network all heading for the same part of town.

In bigger deployments it’s a cluster of intermediaries with similar traffic patterns to any type of proxy. The only extra overhead is the “routing table” which in message land means subject subscription maps. Yes, in theory we probably shouldn’t be copying so much bits, but for practical systems that ship has sailed long ago.

Reply on news.ycombinator.com