Hacker News Reader

Comment by eropple | Hacker News Reader

The difficult part with AWS is local development. Waiting on AWS deploys is not a fast process and it kills momentum.

LocalStack tries, but IMO it's not great.

I agree; this is a rough part. There's always a rough part, no matter how you're writing your app.

I think things like the AWS SAM or LocalStack do a decent job at this. But, maybe more importantly, it requires a shift in developer thinking. Screw local development; yeah, I'll invoke locally during prototyping, but once the essential structure is done I'm going to deploy it and test directly on AWS.

I don't find the feedback cycle due to deployment time to be a blocker. SAM can deploy a function in 10 seconds. Its not an instant code reload, its not ideal, but its also not context-breaking like a 2 minute deploy would be (to be fair, SAM is capable of 1+ minute deploys, but that's usually only when you're adding new AWS resources like new buckets or event invocation rules, not when you're just changing the function code).

Since they added local testing support with `now dev`, I've enjoyed working with Zeit Now (https://zeit.co/now) pretty well. They have the easiest testing story amongst lambda platforms, in my opinion.

Cleanly separate your logic from AWS dependencies using interfaces / protocols, summarily mock the required service by implementing said protocols (sparing you the pain of mocking the AWS services themselves), and voilà, you're running everything locally.

It's easy. But if that's too much for you, the serverless framework has some nice plugins for all of it.

One of the smartest, easier things you can do early on during the development of a function is to treat the "core" of the function body as interface-agnostic. Write a function that just takes an object and returns an object, then write an Adapter function which accepts, say, a lambda event, calls your core handler, and returns whatever your lambda function needs (API Gateway response, etc).

This enables you to, with a little more effort, swap out the Interface Adapter part with, say, a CLI. Or, if you ever want to get off lambda, its a bit easier as well.

Mocking out dependencies, like S3 buckets, isn't worth it during a prototype/MVP. As time goes on, sure, go for it. But early on, just use AWS. Don't use localstack or any of the other various tools that try to replicate AWS. They're all going to get it wrong, hopefully in obvious ways, but usually in subtle ways that'll crash production, and you're just creating more work for yourself. Just use AWS. Just use AWS. Just use AWS.

The agnostic core part is pretty much what I do. Mocking the interfaces is just done to put together the different cores in integration tests and check that the system itself works, independently of what it relies upon.

Then, everything is once more tested using AWS. I stay away from replicating AWS services locally.

It greatly simplifies refactoring and overhauling the core itself, as well as trying out new approaches.

I understand what one can do to create a leaky abstraction for local development. And it is leaky, and it can't be relied upon for correctness, and so while it makes "getting things done" possible, it makes getting things done correctly much more difficult than it needs to be.

And local development when you've opted to use systems like Cognito are well beyond its scope, unless someone has something very clever that I haven't seen.

From a lot of experience doing this on AWS and on other clouds, I have learned that it is better to use systems that you can operate, and then use hosted versions where applicable. RDS is great, but it's great mostly because you can run PostgreSQL locally and inspect its brain. DynamoDB, SQS, etc. tend to be untrustworthy and should be avoided unless you have a bulletproof story for local testing (and none of the fake implementations are bulletproof).

I avoid systems like Cognito for that reason! SQS I've found to be ok when used as a backend for an abstraction. E.g. Laravel has a generic queue library that can be backed by file or redis or sql, ans using redis locally with SQS in production worked quite well with this.

SQS has different characteristics from a Redis queue or a RabbitMQ queue. That's the source of a lot of my nervousness around it: when those abstractions break and somebody-who-isn't-me has to debug it.

(I actually have an answer for local dev with Cognito because my current employer already had Cognito when I showed up, but it amounts to "have configurable signing keys and sign your own JWT in dev".)

I don't use Cognito, and all authentication / authorization takes place upstream, in API Gateway.

Our local tests' leaky abstractions don't care about anything happening upstream. They only care about testing our core "agnostic" logic.

Serverless framework has an "invoke local" feature that solved this for me. I too was annoyed at the speed of deploy.

It's not the running that's the problem--it's the dependencies. You can run Serverless on Localstack, there's a plugin for it, and you can run locally while pointing at Localstack for your "AWS" dependencies, but the impedance mismatch remains pretty high.

(Also, getting a bit further away from that, I find API Gateway kinda shifty and I dislike that it's so hard to run something like NestJS inside of Serverless. Doable, there are examples, but it kinda sucks.)

But there is no deployment delay for the dependencies, so why can't calling the dependencies live in the cloud work?

Are you saying you need an environment to test in that is 100% disconnected from the internet? Given the interconnectedness of APIs in 2019, that seems like making things harder than necessary.

For at-work projects, I don't want to/don't have the bandwidth to be responsible for developers not cleaning up after themselves or exploding DynamoDB with a million write requests or whatever. My team has not yet demonstrated that they can self-manage cloud dependencies and I don't have time to do it for them.

For personal projects, having to futz with Pulumi--and, I should note, Pulumi is the one I like--just to write some code on top of a web server just really sucks. Iterating on a heckin' cloud template just to be able to write some code sucks. Waiting (and as somebody who works on devops projects primarily at work, I am very familiar with how long one waits) destroys motivation to work on anything, both at the start of a session and in the middle--like, blowing away all those dependencies when I need to reset takes way longer than `docker-compose down && docker-compose up`. I am also incredibly cheap for personal projects, because I cannot rationalize spending money on something I'm not certain I'm going to ship, and so I can't adequately ensure that AWS is not going to start dinging my credit card for resources.

The issues highlighted in this thread around the need to develop locally against cloud resources while dealing with a litany of dependencies are one of the things we are trying to solve at Stackery.

This post talks about how our team experienced much of what’s in this thread and what we built to solve it: https://www.stackery.io/blog/how-do-we-setup-a-proper-server...

VS Code folks seem to be pretty excited about our plugin https://marketplace.visualstudio.com/items?itemName=stackery...

We’d love to have you give it a try and let us know how it works for you.

This is, unlike most plugs, actually not bad; the tool looks neat. But I am not your audience. I already have a strong grasp of AWS offerings and tools and know exactly what I want; what I don't have is the inclination to do it for development. If I did, I have Pulumi when I need to build cloud infrastructure and I'm allergic to visual tools at pretty much every level that isn't literally making a GUI.

The problem that I have, and it is probably intractable is 1) I don't want to manage developers YOLOing dev stuff around at work and having a gigantic bill show up because a developer randomed something expensive and didn't know/didn't care to monitor it, and 2) I don't want to deal with the expense or the slowness (of deploys and redeploys, and that includes the AWS resources--ever gone "ugh, I need to burn this down and start fresh" on a dev environment in AWS? it takes forever) of using AWS for development on personal projects.

Thanks for replying with your initial reaction. It's good to get your perspective.

Visual tools: If you are proficient at writing CFN templates or have found another tool to do that, cool. FWIW, our customers tell us they find the visualization useful when onboarding other engineers onto a project and then when they have a split screen between the template and the visualization to see how the CFN template is built.

Yoloing Developers: Heh. I'm stealing that phrase. But seriously, this is a common concern and a reason why managing accounts and namespacing for dev/test/staging/prod environments is a big deal. (plug - we do that too) At AWS dev managers get daily reports on the cost of each developer's accounts. While serverless tends to be much cheaper for dev environments than containers or EC2 instances, there are indeed ways to run costs up.

Deploying and redeploying: That's precisely why our CLI enables development against live cloud resources. When you use stackery local invoke, it assumes the function’s IAM role and fetches the function’s env var values. This enables rapid, and accurate, local function development. On the infrastructure side, it's common for our users to make up to 5 or more changes to their architecture in a day. The more you can do with the building blocks, the less code you need.

Thanks for taking a look.

Serverless framework offline plugin makes this fairy easy if you are going fully down the lambda route

Deploying to lambda is literally uploading a zip.

Yes. Do you know what deploying in my local environment, for my own projects, is? It's "control-S". (Because a file watcher reloads the application.)

This is acutely top-of-mind for me because I used to run a devops consultancy and I currently head devops, including release management, for a startup built heavily on AWS Lambda. I am doing my level best to reduce the loop as much as I can for the dev team.

It's still way, way slower than control-S. And always will be.

What is it that makes this a problem? It has literally never been an issue for me.

I test the lambda body locally with a single command. Only upload a new zip when you hit the release candidate stage and that's not frequent enough for the few seconds it takes to upload to lambda to be a problem.

Until you need a database.

And an API.

And a website.

And a CDN.

And...

And if you get as far as CloudFormation I just have one word of advice. Run!

I used terraform instead of Cloudformation but didn't find it too terrible for what I needed. I realize that it doesn't scale very well to very complicated stuff, but we're talking about using the free tier to prototype things. I'm also not convinced that it's significantly easier to develop locally if you need all that stuff.

CF isn't that bad... if you use one of the many abstraction tools like the CDK or SAM.

It's ugly. It has a TON of powerful functionality that the documentation dives into way too early for most people. It has some weird behavior, especially when you get into networking or stateful resources. It's not perfect. But it gets the job done.

I worked at AWS on a service team, and did a lot of work with CF.

CF's limitations are in part due to the fact that AWS services are all independent entities and the interoperability services have tremendous difficulty getting other service teams to do their part.

The CloudTrail team, for instance, went to great lengths to furnish documentation for other teams to configure their API logging; I know our team fucked it up repeatedly because it was always an afterthought.

You can see in CF that some resources are basically useless, and are plainly provided by CF so that the service owner could check a box. Read "Updating DB Instances" here[1] and then look at all the properties that require replacement.

CF's design reflects this reality. It does not try to understand how services interact or what objects are. Rather, a "resource" knows how to set up and teardown and a few other operations.

If you manage resources exclusively through CF, it can handle most simple use cases. Which is the other limit of CF: by design, they're trying to keep their core invariants simple.

For instance, CF offers no mechanism to stage deploys. As an example, the popular Serverless tool has to initially construct a stack with just an S3 bucket, and then issue an update to that to include your actual stack.

Internally, we proposed doing staged deploys (because wound up hacking them together) and they weren't interested. And I think that's because they would rather keep the API simple even if it means users move away from CF; possibly a consequence of a loss leader business model.

So, yes, CF does get the job done at first, but you will eventually find yourself outgrowing it.

[1]: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGui...

Why? Cloud formation is awesome.

Reply on news.ycombinator.com