Hacker News Reader

Comment by bob1029 | Hacker News Reader

I've seen Tibco Rendezvous used in manufacturing. ~300 megabytes per hour of raw log generated 24/7/365 by tools and control systems in a factory setting. Probably on the order of 10k+ participants in the pub/sub network.

If you are running something like a factory where thousands of independent systems need to communicate in some way, this kind of tech starts to look like the only option.

If you are orchestrating the concerns of 5-10 services, I think you are making your life harder than it needs to be with ESB-style abstractions. Direct method invocation is much more reliable than whatever any one of these vendors could ever sell you. Put all your services into one exe. If you can't be bothered to use one language/repo, there are still ways to achieve this.

The real architect's dilemma is shoving one's edifice-constructing ego into a box long enough to produce a useful shack for the business.

View on HN

Our prod cluster generates that about every minute at O(1M) qps.

We JUST turned on remote Logs because until now Kafka didn't have capacity.

TIBCO Rendezvous is tech from 1998/99, millions of messages per second didn't exist at the time. Only NYSE and NASDAQ were capable of producing millions of events back then (still not by minute nor by second).

TIBCO Rendezvous was one of the first successful large scale, low latency and near real-time pub/sub implementations, and it had a very efficient, Avro like, wire level serialisation format that made messages very compact and efficient to deliver. It was very popular in finance, banking and manufacturing, and is all but legacy now.

$$$ not capability. We have ~50 hosts that generate up to TB per day in just logs and 50k hosts that generate O(200mb/day). The large hosts ssh and grep works surprisingly well, but the smaller hosts is the real benefit.

Hard to justify a team of 7 burning several million in just Logging infra costs.

My previous company's Kafka cluster was handing 20 million messages per second 5 years ago, and dozens of petabytes of data per day. Maybe your particular cluster that didn't have the capacity to handle 1M qps, but Kafka easily had that capacity years ago.

I have to ask, what value is this adding business-wise to store so much?

Not OP but I think it isn't always about storing, but having a log of events which get routed, processed, and aggregated in many cases.

Kafka when used correctly is the like the nervous system for your entire company. You use it like a message bus and dump every single thing into it, and you can extract it out at your leisure, but mostly real-time. It completely transforms how you do services in a company, but it also means you have to invest a lot of money and manpower into maintaining it because it is mission critical.

It was quota and hardware, not ability. This is a single service onboarding and they need the hardware.

And at that scale, we need to grep the logs so the downstream need the ability to process that volume, which it couldn't until about 2 years ago.

NATS was developed by an ex-TIBCO engineer.

It’s not apparent as a dilemma until the said architect has spent years -convinced- that the grand edifice is “good architecture”, and finally matured as a practitioner. Only after that phase passes is there an actual ego-driven dilemma, strictly speaking.

I use NATS to create a secure pipe between buildings and the cloud. I don't need speed but I do need routing and I do need a security layer. Nats just works and took very little setup.