"We will continuously keep an eye on different messaging and streaming systems in the ecosystem and ensure that our team is making the right decision for our customers and Twitter, even if it’s a hard decision"
Rather than setting industry direction, they are inclined to changing underlying platform every few years, which brings a lot of challenges. May be it creates work for platform teams. But, its not useful to do this often.
They had a game changing tool in Apache Storm...once Nathan Marz (creator of Storm) left twitter, another senior engineer took over that group and created his OWN variant of Twitter Storm called Heron. After much engineering PR and github fanfare [0], couple years later he leaves to start his own company to do 'enterprise heron' [1].
So is the goal here to use something that works well for their end-users? Or is it to create a shiny new tool that they can spin off into a new company?
0: https://news.ycombinator.com/item?id=11770758
1: https://architecht.io/streamlio-founders-on-why-the-world-ne...
If the way to get promoted is to launch a shiny new product, then your most senior people will be the best at finding shiny new products to launch, even if that's not the right technical decision to make.
Pulsar is a classic Pub/Sub system. It offers persistence but the basic concepts are what you'd find in most any Pub/Sub system.
Kafka is a distributed log (data structure, not tracing) system. Its basic concepts are what you would expect around something that is persisting events. In many ways its better to think about Kafka as an event datastore and not a messaging system.
In particular, Kafka is much better in use cases where you want to provide all events to many different consumers on their own time frames, over and over again. Use cases like building distributed caches and replication formats will likely fit Kafka better than Pulsar.
Also, Kafka has much higher industry adoption. You are much more likely to have third party support for Kafka than for Pulsar. Also each has a seperate Apache project dependency. Zookeeper has... warts, but they are well known and operating Zookeeper clusters is a much more common skillset than operating BookKeeper.
Again, I really like Pulsar but claiming it is better in basically every way is just false.
Pub/sub is the interface to the underlying log storage in both, and you can layer an event sourcing, service bus, CEP, MQ, or any other messaging buzzword on top.
However, Pulsar goes further than Kafka. It supports millions of topics, multi-tenant namespacing, more consumer options (exclusive, shared/group), per-message acknowledgements instead of a single offset, non-persistent topics for broadcast or ephemeral messaging, geo-replication, tiering to cloud storage (useful for that event store), and a functions platform for lightweight processing. Pulsar also uses Zookeeper for coordination and has official Kubernetes deployments for easy operations.
You're right that Kafka has much wider industry usage, and perhaps that is a significant advantage for most, but the constant announcement of yet another suite of helper tools by every major company only seems to show that it's not quite polished. Other than community and integrations marketshare, I cannot think of any functional area where Kafka is better than Pulsar. Can you name any?
The additional requirement of BookKeeper then is the only major technical negative I see, leaving the market share argument as the major negative. I think saying that Kafka has a lot of helper tools being a negative is pretty unfair though. Pulsar has such lower adoption that there just isn't the rationale for creating the helper tools. That shouldn't be used as metric to decide which is more polished.
So, you cannot have exactly-once delivery consuming from a topic and producing to another topic. Kafka starting from 0.11 supports this.
Pulsar is working on Presto integration for querying which should be interesting.
This presentation compares the Pulsar Function approach against Kafka's 2PC: https://www.slideshare.net/merlimat/effectivelyonce-semantic...
The relation Pulsar ~ Pulsar Functions appears similar to Kafka ~ Kafka Streams. One is a message broker, the other a streaming engine.
Exactly-once (or effectively-once) in Pulsar Functions is depended on assigning unique sequence ids on the producer side. When consuming from Pulsar topic those can be taken from the consumed message, but AFAIK Pulsar doesn't assign those unique ids on its own, so it needs to be done externally.
This is true but I’ll wager that 99% of real world deployments of Kafka are in the simple pubsub use case. Most people just aren’t operating at LinkedIn scale or complexity.
Re: motives,
“Heron grew out of real concerns Twitter was having with its large deployment of Storm topologies. These included difficulties with profiling and reasoning about Storm workers when scaled at the data level and at a topology level, the static nature of resource allocation in comparison to a system that runs on Mesos or YARN, lack of back-pressure support, and more.”
“Although Twitter could have adopted Apache Spark or Apache Flink, that would have involved rewriting all of Twitter's existing code.”[1]
[1] https://www.infoworld.com/article/3078134/analytics/had-it-w...
I just question the intentions when they do a huge PR campaign for an internal project...then the leader of that project leaves to start his own company shortly after.
And no...more PR about heron is not going to convince me that 'it's a huge improvement over Kafka'. Kafka has massive adoption, and almost no Kafka users I've come across have expressed a burning desire for Heron or Pulsar.
One of the oft unspoken things in software is that we let the talent do things that are prima facie suboptimal for the organization in order to keep them here and keep them vibrant.
Some minor boondoggles I used to think harshly about make more sense in hindsight when I think about my own mental processes when I leave a place or even when I’m just feeling demotivated.
I love to solve a problem and be done with it. Probably more than most. But... you have to keep moving, and it’s probably even true that there can be no sacred cows.
At twitter's server spend it makes perfect sense to make such an investment.
I wouldn't link the two problems (size, bad architecture). Kafka is often used to bail out poorly designed data architectures, but it doesn't matter whether the volume is small or large at that point: it's still a pile of crap.
Even for smaller cases it's a nice way to decouple producers from multiple consumers. I wouldn't suggest building Kafka infrastructure if you don't also have at least one bigger use-case, but if it's there it's convenient. Especially if you need replay or the ability to consume the same messages for days afterwards.
https://downloads.ctfassets.net/oxjq45e8ilak/6eh2A72b4IyWsWO...