Amazon SNS FIFO – First-in-First-Out Pub/Sub Messaging | Hacker News Reader

Amazon SNS FIFO – First-in-First-Out Pub/Sub Messaging

aws.amazon.com

34 pointskiyanwang12 comments

Anyone have a good guide to pubsub for CS students? I need data structures and algos, big o complexity. Most guides focus on implementation trivia.

if you could please explain what pubsub has anything to do with those?

I'm a little surprised at the 300 TPS limit. It seems to me that they could have gone with a more Kafka-esque (hah! always wanted an excuse to write that) approach where the write limit is only relative to the message group/ordering id, i.e. a partition.

The thing Kafka doesn't handle so well is increasing the number of partitions dynamically, so in some sense this is just a different name for the abstraction - you could still create N topics upfront and it would be similar to Kafka. But then why have the message group id in the first place if not to allow the topic to scale beyond what a single node can handle?

Is this Amazon's response for Kafka?

There's some overlap in use case, but Amazon already has a more compatible managed Kafka service called Amazon MSK.

I would have assumed Google's pub/sub, which released a similar feature a few weeks ago.

https://news.ycombinator.com/item?id=24658132

That, or they both had convergent evolution and came to the same set of requirements, maybe from some big user at the same moment contacting both of them a year ago, which actually feels really likely to me (more likely than them responding to anything).

"You can process up to 300 transactions per second (TPS) per FIFO topic or FIFO queue"

I want to say Kafka can do significantly higher messages per second on a single partition (and significantly higher than that on a many-partition topic). I don't think it can compete for a lot of use cases as a result. If I were in the market for managed Kafka, I think I would just go straight to Confluent.

> providing strict message ordering and deduplicated message delivery to one or more subscribers.

this is hard yo

I still have nightmares about rendezvous pubsub.

Kafka and SNS only conceptually intersect, SNS is more for intra-service and endpoint notification at huge scale (millions of subscribers) and is directly integrated into a variety of AWS services (e.g. S3 event notifications). You can set up a topic, subscribe a customer via SMS and send a message in just a few lines of python:

    $ python3
    >>> import boto3
    >>> sns=boto3.client('sns')
    >>> sns.create_topic(Name='DopeService')
    >>> sns.subscribe(TopicArn='arn:aws:sns:us-east-1:xxxxxxxxxxxx:DopeService',Protocol='sms',Endpoint='+18008675309')
    >>> ...<repeat for other SMS #'s>...
    >>> sns.publish(TopicArn='arn:aws:sns:us-east-1:xxxxxxxxxxxx:DopeService',Message='sup')

If you're a masochist you can have Amazon notify everyone on the topic every time someone uploads to your bucket

    >>> sns.set_topic_attributes(TopicArn='arn:aws:sns:us-east-1:xxxxxxxxxxxx:DopeService',AttributeName='Policy',AttributeValue='<policy>')
    >>> s3.put_bucket_notification_configuration(Bucket='dopebucket',NotificationConfiguration={<notification config>})

You send directly to HTTP/HTTPS service endpoints, other SNS topics, Lambdas, SQS queues and mobile app push notifications. SNS is more about signaling than a raw message pipe. Relative to Kafka it's basically apples and oranges.

SNS is intended for different use cases. Kinesis and other AWS products provide services in line with Kafka.

If you don't count Kinesis and their literal Kafka offering, MSK.

Reply on news.ycombinator.com