Hacker News Reader

Comment by otterley | Hacker News Reader

I'm curious about the economics of this design. Real-time stream consumption implies that the consumer is always running, and if you need to run software 24x7, running it on EC2 instances is likely to be far cheaper than running Lambda functions continuously.

View on HN

The lambdas are only triggered when there is data in the stream, so the "consumer" in this case is not always running. Under the hood there is a process that polls Kinesis to check for records but it's completely managed by AWS and you aren't charged for it.

Plus, in any real world scenario, if you're running 24x7 your load isn't evenly distributed throughout the day. Which means you're setting up autoscaling (w/ added time+complexity) or provisioning for peak, wiping out your cost savings.

In my experience if your data volumes are low enough like in the article, a Kinesis+Lambda setup is stress free and quick to implement. That makes it worth the cost over raw instances.

It also depends on what your business problem is.

If it's just doing simple ETL then it's probably OK, but if you need to do aggregations on the data, you're going to have a bad time, or end up implementing some sort of ersatz map-reduce framework in lambda.

Yeah, it all depends on how much load or incoming records you expect. In our case, we use the pipeline to import inventory, say images from from our partners. So, we might have a few million images coming in many times a month. Lambda is super efficient in this case.

Depending on the requirements, one can run the sink (we use Kafka Connect for bridging between Kafka and Kinesis) on-premise or as Docker container in ecs or on ec2. That gives you a lot of flexibility to optimize the parameters like costs, throughput, liveness, etc.

Edit: fixed a comma.

Reply on news.ycombinator.com