Hacker News Reader

Comment by insanitybit | Hacker News Reader

> Supports a unique multi-hop network topology leading to a distribution tree for each queue, thereby leading to network bandwidth savings for certain use cases.

This is very interesting. One of the benefits of Kafka is partition routing so I'm curious to learn how this may compare.

View on HN

No idea, but I am interested too. One of the reasons this was probably created was because of a network constrained environment where data is expensive in more than one way.

Cloud Providers can be expensive.

Bloomberg (who built this) operate a system that makes money by transmitting the same data to a huge amount of clients. Having one server transmit the same piece of information to a million client is approximately 500 times slower than having one server transmit it to 1000 proxies who each pass it on to 1000 end users.

The actual architecture of most of these systems in finance relies heavily on UDP multicast, which is a technology that big tech has basically forgotten because it can be tough to administer at large scale.

Multicast puts the entire load of distributing the data in its natural place: the network. The pub/sub queues that the rest of the world uses are more complicated and a lot worse.

This appears to be the beginning of a "cloud-native" bloomberg that can't lean on multicast.

UDP Multicast relies on constantly throwing out the entire state of your system in order to be reliable, whereas with TCP you only need to throw out changes. This is fine in finance because the state is changing constantly anyway so you might as well just continually throw out the multicast traffic.

Message queuing is a more complicated system, but it allows you to push the bulk of your work out to the distributed network instead of concentrating it on the producer.

That is definitely not true about state management - you can build very TCP-like systems on top of UDP (see QUIC), and the final adaptation you need is to figure out how to make that work with multicast (a non-trivial adaptation, but possible). Once you do, you have the foundations of a pub/sub system in the form of a one-to-many (or many-to-many if you are brave enough) network protocol, and now you basically only have to figure out how to do state recovery of new clients and clients who are behind, and that can basically just be a database. The system described in the OP is kind of a combination of a database and load balancer in order to do this over unicast connections.

You have to do this state management with a TCP-based system anyway - the use of TCP doesn't magically come with the full state of past messages.

This is not right in finance at least. Messages updates are typically incremental just like in TCP. The reliability problem is dealt with with careful network stack tuning, core affinity and out of band recovery.

Check out 0MQ (ZeroMQ), Informatica Ultra Messaging (formerly 29 West), and aeron.io They create reliable uni/multicast protocols over UDP, including strategies for NAK storms etc.

Keep in mind Bloomberg works with a few “cloud native “ consumers. In that context it may make more sense.

Well that certainly sent me down a rabbit hole. Seems like UDP Multicast is really IP Multicast. I'd love to play with this

Yes, UDP on top of IP multicast - UDP is a pretty thin layer on top of IP in general, and the IP protocol is deceptively powerful. You just can't run protocols like TCP over IP multicast.

> Cloud Providers can be expensive.

Do you have a story around this? If so, would love to hear more about it.

Specifically, egress from Cloud Providers (transmitting data from inside the cloud to a computer outside the cloud) is the usual culprit for being "very expensive". It's part of a pricing strategy by cloud providers that encourages folks to put all their infrastructure inside of the cloud and to have none of it outside the cloud. As one example of the ink spilled over this topic, see this discussion from 2 years ago about AWS's very high egress fees:

Article "AWS’s Egregious Egress" (2021) https://blog.cloudflare.com/aws-egregious-egress/

HN discussion: https://news.ycombinator.com/item?id=27930151

Ingress can also be costly especially if there is a steady state of high load traffic sent from on-prem machines to machines inside cloud (not sure about aws but have experienced this with azure where we had to resort to buy their expressroute which was very costly and ultimately unsustainable for us)

Ingress is free for AWS. Doesn’t make any sense to charge for ingress…

Something else I was caught out on recently is cross availability zone traffic, especially for multi AZ kubernetes where the nodes are very chatty. It's possible to get the services to consider network topology to limit cross az traffic, but it's not the default.

Can anyone give an example in which there would be less network bandwidth? This tree topology seems like it would use more bandwidth.

Instead of a the tree structure, think about it as Data Center -> WAN -> Site Server -> LAN -> Client

By having a proxy/repeater/site server, you are reducing the amount of WAN traffic you need to send.

maybe if you're hosting replicas or proxies outside of a highcost bandwith area?

low cost end replicas at the end. if the bandwidth usage is half the cost in their multihop topology, 14XXXX compared to 16XXXX? depends on the number of proxies or replication and pricing.

Reply on news.ycombinator.com