Hacker News Reader

Comment by vp8989 | Hacker News Reader

AWS should fork Pulsar and put out a v2 streaming product. Kinesis is kind of crappy (IMO) and doesn't seem to be improving much. If you look at the Pulsar architecture and feature set you can tell that it was designed very much with this in mind (something that large scale cloud providers can integrate with their infinitely scalable storage and compute systems).

It's not all hype either, according to this post https://jack-vanlightly.com/blog/2018/10/21/how-to-not-lose-... it seems like a solid piece of tech.

View on HN

Instead of Kinesis we you AWS MSK (Managed Kafka), which is expensive but works quite well.

Calling MSK expensive is maybe not being clear enough. It's more than 2x the cost of the raw EC2 instances. Then on top of that, to get metrics at the per broker level there's an upcharge, and a further one for per topic metrics. These are things that are actually extremely important at scale, and it's absurd how expensive it gets.

Then you combine that with how immature the UX is, and it really just doesn't feel good to deal with. It's not that much work to run a Kafka cluster with the sort of design that MSK provides, and it can be done better than that without much cost.

How is Kinesis crappy?

I don't think the platform or the pricing model were well architected for a 1 producer, many (and growing) consumer use case which is IMO the most compelling use case for a streaming system.

All the "success cases", sample architectures and real in the wild systems I've seen built on top of Kinesis have 1-3 consumers max.

They added the enhanced fan-out to try to get around this but it seems like you have to (over)pay for a ton of provisioned capacity to ensure you get decent latency on a > 5 consumer use case. So much like Lambda, it's only "serverless" or non-provisioned for people who have super low expectations of what system software ought to be capable of. For everyone else, it's just a mediocre expensive provisioned solution.

Reply on news.ycombinator.com