Comment by Rapzid | Hacker News Reader

Comment by Rapzid | Hacker News Reader

We run quite a bit of infrastructure via AWS lambda via the Serverless project including a data crawling/ingestion pipeline, data cleaning and enriching, and even a flask API..

I agree with the sentiment of the article, if not the specifics of every point. Definitely feel some pain in areas. Sometimes I feel we are doing architectural contortions. Pain points include:

* SQS is not an event source :(

* Cold starts can be very problematic for latency sensitive scenarios when using languages/frameworks that startup slow. It's pre-fork without the fork :(

* No concurrency in event handling exacerbating some of the above

* Artifact size limits can often be rough

* Bizarre artifact storage limits and little help in the way of cleaning them up

* CloudFormation limits and bugs

* The service is rather opaque

* Onboarding; good luck :)

Learned the hard way:

* Created too many individual project repos and serverless services. Classic monolith vs services pains but more to do with too-small service boundaries

* Interface to serverless projects too fine grained(too many lambda functions per service)

* Didn't start using step functions earlier to bridge the gap between stateful processes and stateless processors

* Used SQS as database(long story)

* Ran a flask API as a lambda. Works, but there is just no way we would use this if it weren't internal due to cold start/scale latencies with bursty traffic.

> SQS is not an event source :(

I really don't understand why this is the case. Every time we need a queue for something we end up having to explain this again to whoever our newest developers are, which seems to me like a sign of how unintuitive it is.

I have yet to encounter a use case where we didn't want to grab items out of the queue relatively frequently or even as they were added, so why force us to mess with scheduled events and long polling to do so?

SQS is probably carrying design baggage from when it was introduced in 2004. I've never used it, but would Kinesis be better for what you want to do?

In most cases I think SNS is closer to what I want: multiple producers feeding into a single pipeline that notifies its consumer as soon as possible. The main thing from SQS is the ability to retry a failed message.

If SQS were event based, you could potentially try to process a message as soon as it was received and then have a batch job to reprocess any messages that have been left in the queue however often you like. The closest to this we got was by using SNS and feeding it into SQS, or even a table somewhere, but that gets sort of messy pretty quickly and didn't seem worth the extra infrastructure.

Edit: I didn't really respond to the suggestion of Kinesis here. I haven't done much with Kinesis, but I get the sense it's overkill for the use cases I'm thinking about. I don't plan to have a large amount of producers and don't expect them to have a consistent or large stream of data. If I'm wrong in that being the main use case for Kinesis please correct me.

I haven't used kinesis myself, I've just heard that it's "SQS with the FIFO issue fixed", and wondered if it might also be suitable for you. Judging from the other comments, it wouldn't be a silver bullet, though.

Speak of the devil - I've just had to create a new SQS queue just now (first time in months), and there's an option to make it FIFO.

Ahh kinesis has crazy gotchas depending on what you're doing. If you are pushing LOTS of events it kinda makes sense but see: https://brandur.org/kinesis-in-production for some gory details.

In particular: "you get 5 reads". Number of consumers impacts latency, and not ms of latency, but s of latency.

This stuff is not obvious until you really try and use it.

The API has tons of gotchas around pushing lots of events too; particularly from many producers.

New programming test idea for hopeful candidates; create a Kinesis client that reliably publishes batches of messages and doesn't starve other producers.

You can keep the Lambda function warm by setting up a Cloudwatch event that pings the Lambda function every 5 minutes.

I believe there is a common misconception that this is an actual solution to the problem of cold start latencies.

You can attempt to keep between 1 and X instances of a lambda function running, however the underlying provisioning system is mostly a black box without published details and supposedly not entirely deterministic. Keeping a single instance of the function running isn't going to give great control over the tail on latencies. This is particularly true when faced with bursty, inconsistent traffic patterns.

Hopefully the warm/cold start problem is something they will find a solution for in the end.

Well, Docker already has (experimental) support for CRIU[1]. Since they control the environment, it should be possible to prevent people from doing stuff that would prevent it from working.

[1] https://criu.org/Main_Page

And now you're back to trying to manage the infrastructure, over which you only have very indirect control.

We've come full circle, no?

> Used SQS as database(long story)

I would love to hear it. I have mistakenly used SQS for medium-term storage of state, which is what I am assuming your issue was.

That's the gist of it; tracking position of a long running process by writing messages back into the SQS queue. Any of the numerous edge cases that will cause a dupe record to appear(some of them documented SQS implementation details) would cause the process to duplicate :| Then triplicate :|

Reply on news.ycombinator.com