Ask HN: Why should one ever use AWS Kinesis when Kafka is out there? | Hacker News Reader

Ask HN: Why should one ever use AWS Kinesis when Kafka is out there?

8 pointsamirathi6 comments

I mean apart from the fact that Kinesis is a hosted service and Kafka would most likely require you setting up a cluster. Are there any other technical advantages in terms of features, performance, handling volume at scale, stream ordering, authentication etc.?

View on HN

Another major upside to using Kinesis is how seamlessly it integrates with AWS based consumers such as S3/DynamoDB etc. On the technical side of things, for Kinesis "each shard can support up to 5 transactions per second for reads, up to a maximum total data read rate of 2 MB per second and up to 1,000 records per second for writes, up to a maximum total data write rate of 1 MB per second (including partition keys)" (from AWS documentation), i'm not sure if a hard limit exists for Kafka. W.r.t stream ordering kafka preserves ordering of messages within the same shard i guess which Kinesis also preserves. Again authentication is made easier with the rest of AWS provisioning (IAM) so that's easier to build into Kinesis based application.

I think the bottom line is if you've other AWS components and are willing to be less hands-on about tuning with ease of setup and scalability, Kinesis is your best option. Else go with Kafka

>> I mean apart from the fact that Kinesis is a hosted service and Kafka would most likely require you setting up a cluster.

AFAIK that's the only we use Kinesis over something like Kafka or Spark (much to the chagrin of of our data team). We are planning on moving to Kafka I believe but in the meantime while the Data folks sort all that out we are using Kinesis.

Hosted vs unhosted really.

I mean you can ask, why use EC2 instead of your own machine at the local data-center?

Managed services typically already have taken design considerations for failover, DR, and scaling, so you don't have to.

Well, you can get hosted Kafka as a service as well. My company Aiven offers one such service in AWS/Azure/Google Cloud Platform, DigitalOcean and UpCloud (https://aiven.io/kafka). CloudKarafka is an another vendor with managed offering and Heroku too just launched Kafka service as well.

I think Kinesis is an excellent service, but if you have an existing stack or preference on OpenSource solutions, it's good to have choice.

Hosted Kafka by a third party is not equivalent to (potential) Hosted Kafka by AWS.

There's a lot of signalling/data at the back when AWS implements it, which you will never have access to.

Not a kinesis expert here, but think the need to reprocess stuff could be the key.

With kafka you can reprocess data just creating a different consumer group. You pay for instances.

I guess in kinesis you need to move data from kinsesis to s3 and back to kinsesis to reprocess it, and it wil surely have a cost.

Reply on news.ycombinator.com