Instant scale with AWS Lambda (2017)
aws.amazon.com
aws.amazon.com
Too many times have I drank the Kool-Aid for `someManagedAWSService` and then been surprised by the resultant costs. For example, be very careful when using things like DynamoDB and Lambda. One can easily use DynamoDB incorrectly and end up wasting lots of money. Similarly, I think it would be very easy to adopt some sort of "replace everything with Lambda" approach which would be a misplaced effort as well. Lambda has a specific time and place, it is not a silver bullet.
Not sure what it is. I went with Digital Ocean, but even compared to Google Cloud or Microsoft Azure, Amazon's console area intimidates me, like I feel like I might get charged or trigger charges that will just rack up if I click on anything there. Maybe I'm just being paranoid, but I don't have the funds to risk playing around with their services and accidentally getting billed $100,000 or something.
I guess there's always the risk of someone hacking you, then eating up your AWS resources to mine coins. This could rack up a huge bill. I think this hacking risk is true for many cloud services, though. One way to prevent this is to use IAM users with limited resource usage, and basically freeze the root account. AWS is a lot less stressful to use when you assume a "nobody allowed" or "least privilege" approach.
Sorry for the edits, I am drunk.
Second, they pretty much only charge per resource usage: CPU time, network traffic, and storage. Consume neither of those, and it is free. So it is free to set up really complicated virtual networks in the VPC part of the console, but you start paying once traffic flows in that network.
The only insanely expensive part that you'd hate to enable is the "Dedicated IP Custom SSL" option in CloudFront, because that is $600 per month, as per their documentation[0].
> Because of the added cost associated with dedicating IP addresses per SSL certificate, we charge a fixed monthly fee of $600 for each custom SSL certificate you associate with your CloudFront distributions, pro-rated by the hour.
In sandbox accounts we provision for developers, we use AWS Orgs and service control policies to ensure IAM users with full privs can't provision an HSM.
One could easily configure lots of things wrong in life that would end up wasting money! That said we find that many customers end up saving a lot by moving workloads from more traditional "serverfull" models to Lambda and serverless application patterns. We also see entire startups starting this way and getting to fairly massive scale without ever needing to manage a host anywhere. While not a silver bullet, the use-cases for Lambda are many and proven. More here: https://aws.amazon.com/lambda/resources/#Customer_Case_Studi...
However, the way that AWS charges for throughput on DynamoDB is deceptive -- you can easily bottleneck the system with a hot spot, yet you would need to pay for bandwidth increases everywhere to raise the limit for that single hot spot. This lack of granularity seems to be due to DynamoDB's technical design, and/or lack of a better charging system.
https://news.ycombinator.com/item?id=14721920
I remember reading an article on how some company ran into a bunch of problems due to their non-uniform query distribution, but I can't find it. I am not an expert, but I would avoid DynamoDB today and stick with something like Cassandra or MongoDB for general NOSQL work. Hopefully DynamoDB has changed since I last encountered it as the idea is a very good one and fits well in the serverless ecosystem.
Surely this is not the best use of the service, even if it is common.
If you need to run tasks on a schedule what’s wrong with cron? Or a cron service?
[0]: https://docs.aws.amazon.com/lambda/latest/dg/tutorial-schedu...
I know it works and it fills a gap for people but it just feels odd to me. It's like there is cron -> log -> monitor -> alarm -> event -> lambda -> script when you could just have cron -> script.
I'm probably just hung up on the naming and the fact that these events are branded as part of Cloudwatch. If there was some service spun off that just generated events in AWS I wouldn't raise an eyebrow, even if under the hood it was exactly the same thing.
Lambda has been sold on "saving costs" but what seems to be often neglected is the developer time spent re-jigging into a "serverless" model. Developer resources are far, far more expensive than mere EC2 instances.
I've had issues with the temp memory. It is completely unreliable, you have to store files when processing in /tmp/ and for whatever reason, reading from this results is patchy. I had to resort to storing every file I process in the RAM.
The tooling is only just taking off now, 6 months ago simulation was painful, at least there is AWS SAM now and docker support.
However despite the pain, it's giving me a solution for scalable video encoding that is very cost competitive and I don't have to worry about submitting jobs and monitoring resources.
Lambda is designed to be used for tasks like that, such as resizing images, parsing file etc.
But what I mean is that its clearly not designed for you to be dropping files onto the local disk.
Video encoding is kind of a corner case in that the size of the data really stops you from doing the whole thing in memory, though the persistance isn't needed in between runs.
Probably what you want to look into is attaching an EBS volume to the job, and doing the work from there.