Amazon Managed Streaming for Kafka in Public Preview | Hacker News Reader

Amazon Managed Streaming for Kafka in Public Preview

aws.amazon.com

62 pointsabd1219 comments

This is awesome. Kafka were super fragile to me. Every version will come with all kind of issues and I have to restart here and there :(.

Right now I'm trying to debug this issue: https://community.hortonworks.com/questions/208764/kafka-err...

I'm not a fan of Kafka though but you sometimes have to managed stuf fyou don't like anyway.

So this is great news. AWS truly listen to their customers I think.

Awesome stuff. Kafka has been out of reach for our small org because we simply don't have the resources to manage a prod Kafka+zookeeper cluster. Kinesis has never anywhere near feature parity with Kafka. Ktables on AWS here we come!

> Broker instances start at $0.21/hr and broker storage is $0.10 per GB-month

Kafka requires Zookeeper, do you pay for zookeeper instances too? Anyway, this pricing seems prohibitively costly unless you really need hands-off approach to your infra. Kafka is pretty low-maintenance and stable in my experience.

From the pricing page: https://aws.amazon.com/msk/pricing/

"You do not pay for Apache Zookeeper nodes that Amazon MSK provisions for you, or data transfer that occurs between brokers and nodes within clusters."

Just to note, the $.21/hr broker is on an m5.large (2 cpu, 8 GB mem), which goes for $.096/hr.

We run three nodes right now on m5.xlarge instances. At $.42/hr for the managed kafka, compared to $.192/hr self hosted kafka, I think we'll keep it self hosted for now...

What about the cost of the ZK nodes and associated EBS?

I find that this is true, but when it breaks, man it breaks REALLY hard.

It does. And until last year even the major support company's containers (for Kafka etc) were not very good, we didn't upgrade a k8s cluster purely because of the state of their stack. This stuff scared me and this is a shocking affair these days. In some ways it's a step back in reliability, and i'd have taken this AWS option in a heartbeat, price-gouging aside.

I highly recommend https://aiven.io/ which has lots of managed services including Kafka. Their pricing might be better at low-end.

Hey alienreborn,

sorry that I answer here, this is totally unrelated to the topic being discussed. I've just seen that you replied me on an old HN post of mine and cannot reply there nor can I find another way to contact you on your profile.

It's regarding the habit tracker https://everyday.app I'm working on. You wrote "Your web-app is the best habit tracker ever created and I used it for a while after I paid for it very early on. :) However, I left it because mobile app is not as smooth of an experience. Currently using habitify.me across my devices but will switch back once mobile app catches up with other apps." and I'd totally love to hear in more detail what was lacking or what do you think I should get fixed to help the app catch up. Feel free to answer here or send me an email at joan@everyday.app.

Sorry for reaching out this way and thank you! :)

This is awesome for what I heard about kafka operations.

Huh interesting. I’ve actually had really good luck with Kinesis, this must be their Postgres vs MySQL thing.

What would be the advantage of using this over Kinesis?

Amazon taking more open source and selling it with no value created for the developers. Maybe Kafka should switch to the SSPL [0].

[0] - https://www.mongodb.com/licensing/server-side-public-license

I would argue that making Kafka more approachable and easy to deploy and use out of box is creating value for the developers, because it helps increase potential user adoption (and therefore increases the potential open source contributor base)

I agree. We started using Redis with AWS' managed offering (Elasticache), then migrated to running it on our own once we were comfortable with it. I can imagine the same path for Kafka use.

operations could be headaches tho. I always think AWS as seperate operation contracto teams for running large scale distributed softwares

Then you couldn't run it on Linux, as you'd be unable to relicense the storage system that is the kernel file system or storage drivers from GPL to SSPL.

Open-source software is free, that's the point. If you wanted to make money with it then don't make it open-source. This entitlement is just silly.

Many vendors make money by selling services on top, and there are at least a dozen that offer managed Kafka right now.

Reply on news.ycombinator.com