Ops in the serverless world: monitoring, logging, config management
medium.com
medium.com
This article sure does a good job of disputing its own premise. There's plenty of Ops going on here. The fact that you have to reinvent a bunch of infrastructure, procedures, standards, and workflow doesn't mean you aren't doing Ops.
"You can scale all you want, servers are not your problem anymore!"
proceeds to build a million dollar business
"That'll be <way too many $$> and you got nowhere to run, now that all those clients of yours depend on our service"
proceeds to pay insane amounts of $$ while rewriting in serverful
Is there any standard in the serverless world as to what they accept and how they work?
Numerous companies depend on s3. Along the way open-source solutions have been built, as well as competitive solution that are API-compatible.
We're seeing the same in serverless. Frameworks that are moving toward multi-cloud, competitive solutions, etc. While not totally portable, you can take much of your code with you. You'd do the same if you had a standard Rails app running on an easy-deploy solution like Heroku and had to move to Digital Ocean. In other words, the potential for vendor lock-in is a problem that we've been created patterns to solve for many years.
>> That'll be <way too many $$> and you got nowhere to run
That makes for a great conspiracy theory but AWS has never done anything remotely like that. Through the history of the service they continue to bring down the price on their various offerings. Not sure what makes Lambda the piece they'd abandon all that for (insert evil laugh) world domination.
Then, of course: "You're using the service wrong, silly, all you have to do is sign up for a few years to a bunch of our services".
proceeds...
Don't be fooled, AWS and the likes are very expensive, not as much as Heroku, but definitely more than OVH for example.
I am currently migrating a client's (valuable) production from AWS to OVH to divide their infrastructure cost by 3. Thankfully it did not depend on any non-OSS parts (RDS becomes a Postgres etc).
The question becomes: how far can you go with serverless until you have to move out, and then how much money was really saved once you paid the price of un-aws-ing?
The author of the article clearly spent quite a lot of human ressources and money to get his serverless setup running (hosted ES, Kibana, etc gets expensive fast).
In the end, I find it hard to believe that this is a real time saver, let alone a money saver, in fine.
Just last year they changed the support model to be a percentage of your AWS spend, as opposed to the previous flat rate. They even had the audacity to dress it up as a straight-up price drop.
https://aws.amazon.com/about-aws/whats-new/2016/07/aws-suppo...
Imagine you're an organization spending 100k/mo on AWS and your monthly support line item goes from $50 to $3000. Oh and by the way, the free support tier cannot create technical support requests.
bwhahahaahaha
turn devs loose in production and you will soon realize stability is a good thing
Nothing about this setup feels lean. Why did they build their own feed technology instead of using an open source or hosted solution?
Why did they use Lambda for everything, and invest a crazy amount of effort into making it work well? Their job was to prove their ability to acquire users, they didn't need to innovate on the tech side by using Lambda for everything.
It feels like a lack of focus. I admire how Unsplash is so clear about what the goal is and where their team can add most value: https://medium.com/unsplash-unfiltered/scaling-unsplash-with...
Engineering time is free, of course.
https://twitter.com/JamiesonBecker/status/802185522139582464
http://www.businessinsider.com/london-based-social-network-y...
Lambda is perfect for one-off or seldom-run jobs. Although no one wants to run servers anymore, with any significant volume, lambda is a very poor choice due to costs that quickly grow literally exponentially over the servers used to run the actual tasks. At scale, autoscaling groups are a much better choice for the money.
Edit: I'm guessing 86,400 seconds in a day and ~30.5 days in a month. If so you're using lambda wrong. You should not keep a lambda running forever rather treat each lambda as a short living function that responds to an API request and avoid calling async operations during the execution of it.
30.5 is the average number of days in a month. (365/12)
86,400 is the number of seconds in a day.
So, in other words, based on the very example shown on the Lambda pricing page, 3 million requests in a month is only 1.14 per second.
A single bare-bones t2.nano instance can easily handle that, but let's pretend that it's very, very spikey... so you need 10 nano's. (Of course, let's also pretend that you're constantly using them all, and autoscaling doesn't exist.) A nano can probably easily handle 100 req/s in even the slowest API framework (and certainly the languages supported by lambda), so that's where the rest of the numbers came from.
So, in other words, a cluster that easily handles 1,000 req/s would cost around $67/month, vs $1,600 and change for Lambda. Their very own example proves that it's far from cheap.
The worst part is that as scale increases, the cost disparity increases linearly. Recipe for disaster.
But, again, it's very useful for small things. (AWS certified Solutions Architect and I don't really touch lambda except for things that aren't worth running a whole server for.)
For scaling a lambda it might make sense to run your lambdas on EC2 using this: https://github.com/lambci/docker-lambda. (But at that point you loose the 'NoOps' benefit of lambda and I can't imagine you'd get similar performance to just running a Node Server or the like.)
Throttling.
Concurrent executions are throttles at a (account) regional level, which means if you have 10 different lamdas and 1 of them eats your entire concurrency limit by executing constantly it effectively DoSes the other lambdas such that they are unable to execute.
This may be fine if you are small, but many companies run shared infrastructure and all it takes is 1 misbehaving lambda to paralyse your entire region.
In all fairness, probably not the author's fault, but something that went wrong higher up in the organization.
If that's true, then it sounds to me like AWS Lambda is being grossly inefficient, handling only one request at a time in a given Node.js process. Am I correct about that?
And if you have an unpredictable workload, that may be the most cost effective and responsive solution.
I am sure it has been done before. /s
It's an amazing tool for some tasks, but there's a lot of hype around it too.
Wouldn't you know up front when you're choosing a stack whether they meet your compliance demands?
No, not always. Good luck with the refactor (complete rebuild?) when the time comes.
The number of actors in industries with particularly notable compliance requirements (like HIPAA) that have internally very weak, inaccurate, ideas of the actual requirements and who discover the pracctical meaning of key elements both long after they've been in force and long after they've relied on an inaccurate understanding is, well, non-negligible.
For basic use cases it might be fine, but AWS ES doesn't support X Pack, lags behind official releases, and I even recall it being more expensive once you setup production level ES domains since master nodes are paid for separately. XPack is huge as far as solving headaches with ES, even for the authentication management alone.
We use Terraform to provision our stack though and it seems that the Elastic Cloud doesn't have any programmatic configuration options. How does your team orchestrate your elasticsearch cluster(s)?
EDIT: I should clarify, our use case is not ELK, it's search over a modest data set.
We're pretty write heavy/low read with a comparatively small data set (~50GB expected over 4 months). I know ES can be tuned for that use case to improve index throughput at the cost of slower searching but I never tried setting it up on AES (I'm guessing it'd go in the CF-Template).
One challenge with elastic.co is there's no equivalent to Amazons ElasticsearchDestinationFirehose. We were putting data into a Firehose which batched, backed up to S3 and accounted for failures via a Lambda but elastic.co is giving you "raw" ES instances, so you need to handle those usecases.
You could probably write a Kinesis stream that replicated that functionality(and from what I understand Firehose came to be because that's exactly what people were doing), but we control the devices uploading to the Lambda, so we had the devices act as the buffer (increased send thresholds, and propagated failures to send to ES to our devices which retry automatically) and sent a duplicate record to an S3DestinationFirehose for backup and replaying data
I'm currently tasked with getting a SaaS application onto Amazon and picking it up as I go along, mostly just from the docs. It's been very hard to find high-level overviews of ways to use AWS's ops tools like CloudFormation, most articles I've found are about one specific API and not about architecturing AWS apps.
We run serverless (which in turn applies the CF templates) on Jenkins instead of locally (which also helps automate passing the correct stage and parameters). I guess the equivalent without serverless would be just be using aws-cli, since serverless doesn't do anything special to the CF templates
There was a great webinar on production deployments with SAM (which you could replace with CF) via CodePipeline, but I can't seem to find the slides for it anywhere. If we "did it again" we'd probably use CodePipline/CodeBuild(We just used Jenkins because we didn't really have a chance to look at other options)
I'd conclude that the author realizes they're paying someone else to do it and is rationally choosing not to employ those specialists in their org.
In context, the quote is:
"NoOps to me, means no ops specialization in my organization — ie. no dedicated ops team — because the skills and efforts required to fulfill the ops responsibilities do not justify the need for such specialization. As an organization it’s in your best interest to delay such specialization for as long as you can"
That's not downplaying the value of ops, IMO.