Comment by aynyc | Hacker News Reader

Comment by aynyc | Hacker News Reader

Parent Full thread

Sometimes, I feel like Apache foundation is like Thanos, collecting all the distributed engines and watch the IT world burn.

From the beginning, Heron was envisioned as a new kind of stream processing system, built to meet the most demanding of technological requirements, to handle even the most massive of workloads, and to meet the needs of organizations of all sizes and degrees of complexity. Amongst these requirements:

    The ability to process billions of events per minute
    Extremely low end-to-end latency
    Predictable behavior regardless of scale and in the face of issue like extreme traffic spikes and pipeline congestion
    Simple administration, including:
        The ability to deploy on shared infrastructure
        Powerful monitoring capabilities
        Fine-grained configurability
    Easy debuggability

I can't wait for my next startup interview. We have a requirement of 25 messages per hour, with 10KB per message, you think you can build the ingestion pipeline using Kafka and MongoDB on a 10 node M5d.24xlarge cluster?

Not just startups. I interviewed with a large financial institution recently and they had their heart set on Kafka. From what I learnt , they did not need Kafka -- the message order was not important, there was only one producer and one consumer and the volume of data was not extra-ordinary -- but they wanted to be on Kafka as that was the enterprise direction. I might have come across as an old fart when I asked the why they could not use their existing MQ infrastructure.

Not that I'm not impressed by Kafka and its stability, performance and scalability, but I see the same behaviour from our customers.

They specifically want Kafka, there's no real reason other than they need a queue, which Kafka actively states that it's not. At that point it gets really tricky to reason with the developers about why they might be better served by something else. Generally speaking it's not much of an issue, because Kafka will deal with workloads just fine, it's just weird. I have seen one customer use Kafka as a database, that works less well.

We do see the same with Kubernetes. The developers pick Kubernetes and at that point it's to late. They specifically want Kubernetes even if you could more easily solve the problem with Nomad, Docker-Compose, plain old VMs or EC2, depending on the problem.

This is the exact same problem we are facing as well. Our clients just want to use it without knowing or thinking what problem they are solving.

Confluent itself says that for a workload of 38Mbps/sec rabbitmq has 5 times less latency than Kafka and is 100x simpler to manage.

https://www.confluent.io/blog/kafka-fastest-messaging-system...

How do you teach someone to look at the problems first and then pick the tools?

> How do you teach someone to look at the problems first and then pick the tools?

By understanding what the person cares about. Everyone knows "pick the right tool for the problem". Not everyone uses such a simple calculus because life isn't that simple. People have their own agendas, backgrounds, experiences, career growth desires, personal lives, etc., that are all part of their personal objective function. If you want to convince someone that your tools are better, show that your tools have a higher payoff for their personal objective function. This is way more than a mere product question. In a team setting it's even harder, because you have to balance it across multiple people simultaneously.

30 not 38. Also, 30K messages/s not 30Mbps.

"Due to CPU bottlenecks, we were not able to drive a throughput higher than 38K messages/s, and any attempt to measure latency at this rate showed significant degradation in performance clocking a p99 latency of almost two seconds."

I'm about to pick Kubernetes even though a different solution would theoretically be a much better fit for my needs. This is entirely because some other tools I'm looking at play nicely with Kubernetes out of the box. If I picked something else, I'd end up writing my own glue code.

I wonder if something similar is happening with Kafka?

My last gig was Kubernetes, and aside from all the hate it gets here, (You're not Facebook, you don't need Facebook scale) it was a very pleasant experience. So pleasant in fact that when I moved on to my next job (Amazon EC2 VMs) it was pretty painful. They were running an old version of Amazon Linux and hadn't been updated in years. The versions of some runtimes were impossible to update due to GLibc being out of date. Our immediate answer was, can we at least get to a Docker solution? ECS/Fargate provided a nice middle ground. But I'll admit. Once you start getting into running multiple replicas, it's nice to have the other stuff that Kubernetes affords you.

I pick Kubernetes because I want to manage software and not manage servers. I think it gets a lot of hate because people look at helm charts that are designed to support all possible software configurations and they are quite confusing. If you break it down to the basic pieces of configuration it's not really any more complicated than say a docker-compose file. Just a little more verbose.

Oh definitely. Helm/Kustomize.... Yamls all over the place. It can be awful. But then how nice it is to have a cluster with load balancing, a nice API Gateway, services spinning up and down gracefully. While that is certainly achievable in other ways, this one has been my favorite. (I used to deploy WARs to JBoss and have zero downtime... while it was possible, it was horrible)

AKA Resume-driven development, the enemy of all engineering efforts and reason

There's also strong pressure to take whatever the "safe" choice seems to be. If everyone's jumping on some technology, and your manager already mentioned it, even, then you will catch no blame even if it's entirely terrible. Advocate for something else, and win, and congrats, now even if it's way better every little hiccup and difficulty is your fault.

To experience it on a smaller scale, get a family member (non tech savvy) who uses windows/macOS to use linux for a few days. Everything that is not the same as it was on windows/macOS will be your fault even if the linux way is better/faster/cleaner than windows.

Framing it like this is unproductive IMO. There's merit to picking technologies that help you and your teammates grow as engineers. You can take it too far...we should accept that it's a messy process to find the balance that lets us be productive now while helping us be more productive in the future.

As an engineering leader sometimes that even means knowing that people are making the wrong decision, and letting them do it anyway, and then helping them learn from it.

>> Resume-driven development Lol . Love it. I shall be using it from now on in similar context.

ha! It's funny because it's true.

> which Kafka actively states that it's not

Where do the docs state that? Might be a useful link to keep handy.

Hmm, that might be me remembering wrong. At least I can't find it. Sorry, I may be wrong.

They do go to great length to avoid calling Kafka a queue. No where does it directly state that Kafka is not a queue. The docs just never talks about Kafka as being a queue.

Summary > Latency section [1]

https://www.confluent.io/blog/kafka-fastest-messaging-system...

It's the same with any new shiny technology it seems.

The specific thing I have experience with is in analytics/relational databases. Suddenly around 8-10 years ago it became imperative for every client I was dealing with to migrate their RDBMSes to Hadoop/Hive setups, even when their largest dataset was only about 120M rows denormalized.

They were trading three servers (primary, backup, DR) for sometimes 15 to 20. Queries that MSSQL was handling sub-second were suddenly taking 45s on Hive. It was utter madness and was as far as I can tell driven by good salespeople, FOMO, and the feeling of importance of being able to say your company is running Big Data(TM?).

I saw maybe one implementation (of dozens) that actually stayed in use for any length of time.

We've been using Kafka for a while. We're thinking about ditching it for something AMQP. For all it's bells and whistles, it's just not really all that exciting. We're probably spending 5x more on Kafka than we could something much simpler.

Kafka is only exciting when you're wanting to stream a metric shit ton of data without things falling over.

If you're using it, you're constrained by its engineering decisions, so you need to be sure it's a worthwhile tradeoff.

We're doing the same. Actively moving away from Kafka, which was only ever used as a message queue anyway, to an actual message queue. We might move back at some point in the future to rebuild an actual event-driven processing model. But for now, it's not worth the additional complexity for the scale and systems we actually have.

Which product are you moving to?

Just to clarify...AMQP is a protocol so you will need a broker that supports it. The two options I see for you are RabbitMQ and Solace. Both support AMQP.

Apache Pulsar seems to have decent AMQP support also.

The main thing Kafka is great at is scaling out via partitions with it's natural stickyness, while maintaining excellent ordering guarantees. This means it's really easy to create a load of reliable in-memory processing where each thread processes a single partition of data.

When scaling out on a standard message queue you generally have greedy consumers which means you can't assume stickyness, or have to create your own partitioning structures. It makes it great for realtime apps...

If there were cheaper alternatives I think people would use them but it does definitely have powerful benefits.

What would you recommend as a "pipe with MQ-ish semantics" to use when "the message order was not important, there was only one producer and one consumer and the volume of data was not extra-ordinary", but you do still want the core premise of an MQ — reliable message delivery in the face of faults in the producer and/or consumer?

We actually have a use-case that exactly matches this. One service makes [stuff], the other service consumes [stuff], both services are "immutable infrastructure" with no local stable state storage, and [stuff] is individually too small and frequent to be affordable with IaaS managed-MQ per-message costs — but batching messages into reasonable chunks before send means potentially losing up-to-a-batch worth of messages if the producer dies.

Not OP, but I would recommend RabbitMQ. It's surprisingly simple to setup and if your load isn't huge it'll hum along just fine without clusters other bells.

> I can't wait for my next startup interview. We have a requirement of 25 messages per hour, with 10KB per message, you think you can build the ingestion pipeline using Kafka and MongoDB on a 10 node M5d.24xlarge cluster?

This reminds me of something I experienced

Back in 2000 I worked at a company that hired me to take over their brand new fully redundant web infrastructure. that was architected and built by a firm on a $10,000,000+ contract.

They couldn't understand why the site only served 1 page every 2 seconds. To be very clear: _1_ page every _2_ seconds.

The short story was the development company didn't have anyone who understood databases. So they were using the oracle cluster as a key-value store, and then parsing/sorting XML files on the application servers on demand.

This was a $500,000/yr oracle license, on redundant dedicated $(million) Sun hardware, with huge high speed disk arrays. They were almost sitting idle at peak load... of 1 page every 2 second.

Bonus story: The developer's didn't understand why their dynamic uploading of files to their app servers only worked 1/x of the time where X was the number of app servers in the cluster. They were dropping the files locally on the app-servers and wondering why the other app-servers didn't know about them.

It took a dry erase board and about 30 minutes before they truly understood that the filesystems on those devices were not shared and why. (note, it was never part of the design specification their own team created)

I'm happy that I didn't have to explain ephemeral containers to them.

> We have a requirement of 25 messages per hour, with 10KB per message, you think you can build the ingestion pipeline using Kafka and MongoDB on a 10 node M5d.24xlarge cluster?

Sure, if you have the budget to run it, can do! Feel free to reach out, email in the profile ;)

> Sometimes, I feel like Apache foundation is like Thanos

Soon: https://thanos.io/

Reply on news.ycombinator.com