AWS Copilot | Hacker News Reader

AWS Copilot

aws.amazon.com

153 pointsbdcravens92 comments

Not directly related to copilot but in context of ever increasing yml/json configurations for cloud apps, am I the only one who feels overwhelmed with the configuration options? I see tons of articles with just drop this json here and that yml there and run this cli command - where is the reference for these json/ymls? Does everyone fully know/understands the structure/options? How do people keep track of the changes to that between upgrades? I feel increasingly dumb the more I see these configs.

There's definitely room for config-viewer plugins in VSCode, Sublime, etc. to help write and maintain the 100's of config files we have to be knowledgeable about in DevOps these days.

Does anyone have good suggestions in this space? Or do I just have to google docs and reference documentation when I need to figure out what the hell all these parameters mean?

I think clouds have been trying to fool developers into thinking that cloud is easy. Don't feel bad, building a reliable infrastructure is incredibly hard. There're ton of services, some of them are redundant, and it takes a lot of time to discover all the nitty-gritty.

Maybe it's messaging intended to be for grizzled old veterans going "look how we've taken the old complicated thing and given you powerful tools to allow you to manage it more easily with less steps" but then gets interpreted by neophytes as "look how easy it is for a complete beginner to come in and set up a complicated system".

Great observation. A lot of messaging for cloud products is comparing against legacy or dominant solutions. A beginner doesn’t have that reference point.

Messaging is written by marketing and sales departments.

Employees there usually don't have depth of technical expertise or experience to draw from, and are incentivized to make the product sound as easy to use as possible.

This combination results in {really complicated thing} -> {simple abstraction}. Ultimately, everyone internalizes this and starts forgetting that {simple abstraction} is actually {really complicated thing}.

Happens with any popular technology. Networking, DB, cloud, ML.

Unfortunately, this flies because the primary buyers are also non-technical.

Oof yes. The times I read stuff like "to make sure your WidgyPidy will automatically balance your BoolyPooly cluster, just set the autoEnableBalanceBoolyClusterGenerated in your WiPoBaClConfig.yml file to the desired floating point and drop a reference in the ServerClusteringPoolDirective.json in your configured ShimpyDimpy config folder" like it's obvious and simple...

hahaha.. lol.

This starts to sound more and more like how they manufacture plumbuses

High IQ joke

I always wondered how plumbuses got made.

Me too! Now I know: https://www.youtube.com/watch?v=eMJk4y9NGvE

This is just.. accurate.

Finally, DevOps has caught up to the rate of progress in js framework land. Rejoice!

Urgh, yes.

I'm not a dev by any stretch of the imagination but deploying PHP apps on servers was significantly easier than this mess we have now.

Rsync/ln is still the fastest deployment process I’ve had so far.

Still too complex for me, let me just login to an sFTP and drop some files in

>I feel increasingly dumb the more I see these configs.

Especially since there is no IDE support.

I've come back to enjoy XML with nice schemas and IDE validation and autocompletion instead of YAML and liquibase changes not going through properly due to an indention problem.

You can use schemas with json/yaml (e.g. jsonschema or protobufs); I wish people used them more though

I feel like trying to use a "dumb" declarative config language for everything is part of the problem.

The domain is not simple enough for that, so you end up with all kinds of weird scripts around it all, or other templating/codegen solutions, plus all kinds of ad-hoc conventions that are extremely easy to break and very annoying to navigate.

Having a full blown statically typed language like TypeScript - where things can reference each other strongly (not just by names and other strings), naming conventions can be encoded in functions (or even enforced by types), and structures on every level (even your project-specific ones) have a well-defined and checked schema - would help a lot in my opinion.

Maybe deno could find a use in this niche.

You're (accidentally, I think?) describing Pulumi in striking detail.

https://www.pulumi.com/

Kind of. From a brief look, it looks very framework-y and all-or-nothing (though I could be wrong, they do have a page about gradual adoption at least https://www.pulumi.com/docs/guides/adopting/).

I was picturing something more low level and less opinionated, though I can only guess how such a thing would perform in practice.

Not even necessarily a specific product, just the adoption of workflows that use already existing tools (e.g. the various API client libraries - for AWS, Kubernetes, git, docker registry, etc.) to achieve this. I realize that that could lead to reinventing the wheel, and eventually an in-house framework anyway, but it feels a lot more comfortable to start there, as you can design things closer to your exact needs.

Pulumi definitely looks interesting though.

It sounds like you're describing HashiCorp's HCL2 and Terraform Configs which are at least a graph.

I find using AWS without Terraform to be nearly impossible.

Terraform, or Troposphere, or cfn-modules/Widdix, or CDK.

Cloudformation as a JSON/YAML vanilla config is just awful, and really exposes the inconsistancies between the different services.

Yep. I prefer to use language files over JSON or YAML files if I can.

...like CDK?

https://docs.aws.amazon.com/cdk/latest/guide/home.html

Have you seen aws-cdk or pulumi? Does exactly that with TypeScript to boot :)

Scripts which generate the config also have their own problems. Check out Dhall, it attempts to solve this.

Reminds me a bit of jsonnet (https://jsonnet.org/) which didn't catch on much as far as I can tell.

I see very little practical value in a DSL like this. Using an existing language with an extensive ecosystem and great tooling was half the point of wanting TypeScript in the first place. Most of the "risks" can be handled with just a few eslint rules, tests (e.g. all outputs must satisfy a set of constraints), and code review.

Especially when the moment you need to do more than just generate config files (e.g. fetch some information from an external service, or even write an imperative script for the occasional task), you'd have to switch to a different langauge and all its tooling - where you can't just import the exact same type definitions, etc.

This also sounds like the Nix language (https://nixos.org/).

For aws, I usually read the cloudformation/api/boto3 docs. https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGui...

I don't know all the options, and I don't need to either. Can look at examples to get started and customise as I need. What changes?

Idk if other ways are better, just looking at large text config files for apache or other linux software boggles my mind just as much if not more. Once you have lots of options it's just going to be complex to understand.

Track changes in version control (git) using infra as code tooling

And get the second order effect of one of those configuration options wrong and wake up to a huge AWS bill. And, yes, with a bit of groveling they'll probably refund you after you've sweated a bit--the first time at least.

Or worse, you fudge a YAML line and bring the application down.

There are tools to catch and prevent that sort of thing: Datree, AWS CloudFormation templates, and Sentinel, just to name a few. They apply and enforce different policies to prevent YAML/JSON (or “infrastructure as code”) misconfigurations.

(Disclaimer: I work with Datree, but not the others.)

No, I'm not overwhelmed by documented/structured config files.

20 years ago I was a Windows admin, discovered Linux, and it blew my mind how much easier it was to configure, experiment, validate, and roll-back config changes. GUIs are nice for discovery and exploration, but there's a very good reason they're not used for server and infrastructure configuration.

> yml/json configurations for [...]

We learned this lesson the hard way with XML back in the early naughts when behind everything was a nauseating morass of xml with overcomplicated designed-by-committee schemas.

People moved out of development and tried hard to forget it, but neglected to instruct the next generation about the pitfalls of shitty configuration files that get out of control.

Now we got json and yaml. No more </>'s, yay, but somehow worse, less maintainable and less understandable.

The new J2EE.

Note also that the Docker integration into Fargate was released at the same time, see https://aws.amazon.com/blogs/containers/aws-docker-collabora... and https://www.docker.com/blog/from-docker-straight-to-aws/ - this provides a simpler way of using Fargate, especially if you are already using Docker Compose. You can output cloudformation from this as well, but you don't have to understand the internals of ECS to use it. This is in beta and would love more feedback (here https://github.com/docker/ecs-plugin); I worked on this at Docker with Amazon.

My goodness, Docker Compose file to production stack?! My dreams have come true.

The arguments for k8s I have is that the local and production setups are identical.

It's a PITA to run k8s locally, but the alternative is that you use Docker Compose local, and then deploy to k8s, which means two sets of configs. And that's terrible.

Or, you use Docker Compose locally, and fragment all of the containers into individual ECS/Fargate services which you write the deploy pipeline for individually.

These were your best options, for one glaring reason: You can't use a docker-compose.yaml in production. (I know, I know, Kompose, but it doesn't always work quite right and you now have a k8s cluster).

Really excited to try this out, thanks for the work on it!

I happened to catch this on the Github daily trending list a week or so back, really cool.

For similar tools (though this one is ECS + Fargate), also see:

https://somanymachines.com/fargate

https://github.com/awslabs/fargatecli

I'm pretty particular to Pulumi, and defining infra in typed programming languages with full IDE tooling. AWS CDK is the AWS-centric equivalent, and that's a great and under-represented tool as well IMO.

https://www.pulumi.com/containers/

https://docs.aws.amazon.com/cdk/latest/guide/home.html

My issue with ECS/Fargate is that on the surface, it seems amazing. And so you deploy your container, and that's relatively easy.

But now you have just a container on HTTP, on a bare IP. You probably want HTTPS and a domain name attached.

So then you learn that you need to:

- Create a Route53 hosted zone for the domain

- Use AWS Cert Manager to provision certificates for the domain

- Create a load balancer, and figure out how to attach it to the ECS/Fargate service

- Wire up the LB address to the record set and configure the termination for HTTPS

And to do something like change an environment variable, you have to create an entirely new "task definition", deploy the task revision, and then spin up the new service and stop the old one. This takes about 5 not-particularly-easy to find or navigate screens in the AWS console.

This is what it ended up looking like, for me to deploy a Fargate service:

https://gist.github.com/GavinRay97/fbedfb18fe852bf625451a417...

It's great that these tools can help reduce the burden of doing this but man is it a painful experience if you're not a wizard at DevOps.

Google Cloud Run is 1-2 bash commands to deploy a containerized app on HTTPS.

Copilot can also help set up HTTPS/ACM Certs/Subdomains:

https://github.com/aws/copilot-cli/wiki/Applications#additio...

I love the cloudrun style service too - but one thing I think is cool about running on ECS/Fargate is that once you outgrow the constraints of CloudRun style services - you can drop down to the more granular infrastructure to make tweaks and adjustments. Just my 2c :)

> Copilot can also help set up HTTPS/ACM Certs/Subdomains

Oh yeah for sure! I was really excited about this for that reason when I found it because of recent Fargate pains.

The irrational part of me just wishes that the "magic" from tools that AWS publishes like Copilot, Cloud Development Kit, Fargate CLI, etc could abstract it away in more of my interactions with AWS ;^)

> once you outgrow the constraints of CloudRun style services - you can drop down to the more granular infrastructure to make tweaks and adjustments.

Definitely, you're trading convenience for flexibility with Cloud Run vs Fargate. Cloud Run also doesn't support websocket traffic yet ( RIP =/ ).

I'm a massive fanboy of Serverless Containers and think they're the best thing since sliced bread. I run services on both AWS and GCP, and think Cloud Run and Fargate are both solid.

My takeaway with AWS I guess is: Don't interact with provisioning/managing infra raw, use the CDK or other tooling you publish haha.

> Don't interact with provisioning/managing infra raw, use the CDK or other tooling you publish haha.

I've moved pretty much all my sandbox exploration to CDK. A one-shot destroy when I'm done to clean everything up is great. I also appreciate the documented types in Typescript. But beyond that, I've found that there's really no easy way to manage permissions manually. Doing ` bucket.grantRead(service.taskDefinition.taskRole);` is so much simpler than any of the policy generators in the aws console.

I believe it's possible to write the ECS task definition as a JSON file, include environment variables, commit it to a repo, then use AWS CodeDeploy/CodePipeline to update ECS with a new task revision whenever the repo changes...

...which kinda proves your point. This stuff is not well documented and it's ever more complexity.

Initial infrastructure setup may take a few extra steps but what really disappoints me about the Fargate platform is lack of inspection capability.

Quite often I want to strace, tcpdump, etc. Cloudwatch logs is a little anemic. Want to see an adjacent line in a log search? Good luck.

It's nice that AWS services tend to do one thing.

But it does sometimes mean that there's always just one more AWS service you need. One more service with a different web console UI, varied support in the different infrastructure tools... and now we have another one of those.

All I seem to do these days is AWS infrastructure, sometimes as code. I'm not sure one more new tool is going to improve that.

Hi!! I’m a developer on AWS Copilot! We’d love your feedback!

Homebrew: brew install aws/tap/copilot-cli

Docs: https://aws.github.io/copilot-cli/

Feedback & Requests: https://github.com/aws/copilot-cli/issues/new

Thanks for your work on this!

Wondering if it’s possible to use this with our own self-managed ASG of ECS Host instances. I trolled through the GH repo quick, but couldn’t find any definitive answer.

Not yet - we’ve started out with Fargate but we have a roadmap item to support EC2 and resource imports:

https://github.com/aws/copilot-cli/issues/1094

It's a pity you used "copilot", because that name is already used for Red Hat's admin product.

Do you mean cockpit?

Oops yes. But there is also Performance Co-Pilot (pcp)

fogcreek/glitch then stand-alone https://www.copilot.com/

I'm actually in the middle of moving some apps to ECS/Fargate. Here are a few of our requirements, it'd be great if you could share if copilot meets or does not meet them.

1. Production and Staging in separate accounts 2. A new environment provisioned per github PR with setup/teardown - ideally within the same VPC/Subnet ranges, so they can utilise shared databases 3. Secrets stored in Secrets Manager or Parameter Store 4. Blue/Green deploys for production with a manual cutover step (manual step optional) 5. Deployment of 2 services (app, background tasks) from the same dockerfile/commit

Awesome - first best of luck on your transition!

1 Yeap! We can do that. 2 hmmm we don’t support this out of the box - but I’d love to know what you expect out of this. All your services spun up and infra? 3. Check! 4. No blue green deployments yet :( 5. Yea! We support this via a pipeline.

Number 2 would attract an enormous amount of developers. Can you consider doing an example of such a workflow with copilot and pipelines, including provisioning a LB and/or tweaking an existing one to get per-branch subdomains with https?

As far as expectations for 2 above, in my experience, teams really love having a completely separate environment per PR. Gitlab has done this on top of Kubernetes using a dynamic domain such as <truncated-branch-name>-review-app.domain.com. This gives QA and stakeholders the ability to easily review/signoff on changes prior to going live and not getting bottlenecked by a small number of QA/UAT environments. This becomes particularly helpful as the engineering team scales up.

For 2 - we would already have the time infra spun up via something like terraform. So we’d need to “adopt” the VPC/subnets from copilot. The actual connection details would be stored as env vars, but would need a “on create” hook to create the database for the environment.

Hi - I'm interested in using this to manage deployments on top of a Fargate cluster managed by Terraform. Is copilot able to import such a cluster or otherwise manage tasks in an existing cluster?

Not yet but we’re tracking resource importing here: https://github.com/aws/copilot-cli/issues/740

Remove the emojis please

This. I almost stopped reading the docs after the first screenshot. Really, it gives a childish feel to the tool. I don't want my tools to have rocket or bear emojis. It makes me feel like the tool was written by teenagers. Of course that's is not the case, but that is the feeling that I have when I see this.

I understand. It’s only on the help page :)

It’s written by some of the brightest people I’m AWS - so Don’t worry about that.

Counter-opinion: I think they do a good job of visually distinguishing the section headers in the help output. As long as they're used structurally like that, I think they're great.

We currently use Beanstalk, which seems to share a bit with this - in terms of offering a CLI they abstracts a lot of AWS specifics, deploying to ECS, the application/environment split, the deploy command fitting in a CD pipeline. Can you share a bit about how your team thinks about Beanstalk and this? We’re quite unhappy with Beanstalk from the perspective of trying to define it with Cloudformation for reproducibility and I’ve been unsure what we should be looking at next.

Do you know if EKS will ever be supported?

The tool looks promising (and of course it is just a couple weeks after I finished rebuilding our ECS CloudFormation from the ground up... lost count of how many times AWS has done that).

My question is a bit less targeted at this specifically and more about how AWS approaches all of these various tools.

We have probably 95% of our infrastructure in CloudFormation one way or another (either directly or with something like SAM).

As more of these tools come out there is going to be crossover (and there already is with Amplify and SAM for example).

There will also be questions about how we pass exports around so we don't have to hard code endpoints that were created by another tool. (Say a Lambda created by SAM needs to be called by a service in ECS).

So my question is a couple things:

1. Does AWS have a requirement or a goal in mind of how to always ensure we can pass these exports around ideally in a consistent way (this and any future tools that may come out)? I worry about having too many different ways that things work from a code standpoint that we just introduce more bugs into our system.

2. Is the thought that a given repo should ideally be using a single tool (I ask since I see you mention S3 buckets, and databases as a coming soon) and not mixing things? Like not using this plus some native CloudFormation for example?

3. Finally, I would love to see a... "deployment tools" page or something from AWS. A page that is a table of every deployment tool for columns and every service for rows and can easily see at a glance what tool can do which services. Right now I am considering if I need to build a page like this for internal use so a developer can easily look and see what fits their use case the best. But a page managed by AWS would be great.

Ultimately what I struggle a bit with tools like this, are what does it get me long term over just using straight CloudFormation where my only limitation is CloudFormation lagging behind.

Not from AWS, but what it sounds like you're asking for is better cohesion and more consistency between AWS products and services?

I've been an AWS customer for over 10 years, and have been asking them for this (on behalf of my enterprise customers). I'm always told "they're working on getting better" but they never really do.

Cloud formation features are inconsistent across products. Documentation is formatted and structured differently depending on the tool. The AWS web console is an ever changing grab-bag of new and old. Some products integrate well, other products feel like you're using an abandoned hack-week prototype.

Part of the reason that AWS is so great is how much it can do, and I think that's been made possible by the apparent level of autonomy the teams have. However, it's at the expense of the customer's experience.

I totally get that, and its for sure the reason I both love and hate AWS. They release things at a near breakneck speed.

I guess its more, if I had to choose one area of AWS that I wish there was a but more consistency it would be their deployment tools. It really just comes down to, when I start something it's almost an agonizing choice of what do I use since I am worried that I could get stuck in a box.

I don't know what the answer is, that is just my view of all of these tools.

I probably missed it in the blogpost and github readme, but is there a good way to "export" this into a CloudFormation/Terraform template? After all, this seems to spawn a few things (networking, mostly), so I want to make sure I can automate any further deployments.

Thanks!

Copilot generates CloudFormation templates based on the Copilot-specific config files. There’s a place where you can add your own additional resources in CloudFormation syntax IIRC. It also has the ability to create a CodePipeline for app deployment. That creation is also done by generating a CloudFormation template and deploying it for the user. I’m not sure why it’s not clearly documented that this is happening, but it is a selling point to me because if I want to, I can “eject” from Copilot and just maintain/enhance the CloudFormation templates.

I have a hard time believing anyone would consider ejecting to CloudFormation a selling point.

Could be nice to version control them. Some CLI stuff looks cool on a blog, but do you want to remember what commands you typed, and in what order, to bring up what production looked like two weeks ago manually?

It's a cautious way to start working with a new tool. It's AWS, but there will still be bugs, especially for people with large or unusual deployments. You could use the tool to generate CloudFormation configs and then put them through a PR process. After six months of this with no bogus configs generated by the tool, you might decide you're on a well-paved road and make it part of your CI/CD pipeline.

Yea! You can run copilot svc package which spits out the CloudFormation for the service.

What are the advantages of this over CDK? It's trivially easy to spin up a loadbalanced service https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws...

CDK is terrible on so many levels, it's possible that this is less so.

Expanding failures of CDK due to popular demand:

- It introduces an edit-test-debug cycle from hell, it doesn't do what you think from the mental model of a user-friendly CF template generator library (doesn't statically ensure stuff it emits actually works even superficially)

- It builds upon an already difficult and leaky abstraction (CloudFormation) so you just get additional failure modes and cognitive burden

- Glacially slow iteration, can easily take 10-30 mins to teardown/setup your app due to CloudFormation.

- fails to provide upsides from the costly tradeoff it makes to jump into turing complete programming languages, like 3rd party cdk components or iterating expressions at a repl

- don't even get me started on the npm dependency hell (you'll randomly be debugging mystic npm warnings already on the next day when installing a new submodule of cdk deps, when version mismatches appear), the other-languages-to-npm bridges, the "you can't use this without an IntelliSense style IDE" API design, buggy cli tools, etc

To summarize it doesn't manage complexity or give you a good abstraction. Probably because it tries to take on all of AWS functionality at once.

(Also the GP linked Fargate example is far from the functionality from the copilot example, like enabling you to actually dev and deploy the app - ecr, codebuild, codepipeline etc)

Hiya! My team is also the team that writes and maintains the CDK construct you linked to.

I think it’s different approach - that’s all. Copilot Is more focused around a full developer experience including setting up CI/CD, deployment environments and operational tooling like logs, stats, etc.

This reminds me of Convox. Very simple to get started and runs on ECS. Copilot has less features as of now but looks like it’s going in a good direction.

(Happy convox customer)

This really seems like it could be a future replacement for Elastic Beanstalk. Especially when they finalize the ability to provision storage infrastructure.

That’d be nice. It was a good concept but I really didn’t like how it handled some of the application commands with yaml files.

Hopefully. Elastic Beanstalk is a product that "works" and is "maintained", but when it fails it crashes hard. At least for the PHP environment, updating the environment variable configuration can take 2+ minutes, and non-immutable app deployments with 2 hosts can take over 10 minutes to pass health checks. And finally, if it fails to do something (eg. a deploy command fails on the instance), it will be stuck in the "error" state for 5-15 minutes where you can't do anything to the environment and must wait for it to clear to fix your bad deploy.

I should clarify that the app deploy itself can take under 5 minutes if non-immutable; immutable deployments are the ones that can take 15 minutes or more.

Amen I thought I was just doing something wrong! What is the best alternative to Elastic Beanstalk on AWS? Is Kubernetes overkill?

I wrote a blog post about this that did the rounds last year about the topic I believe Copilot is trying to solve:

https://leebriggs.co.uk/blog/2019/04/13/the-fargate-illusion...

Summary: the marketing spiel about how easy fargate is to use and get up and running isn't actually how the experience is. I'm hoping Copilot improves that experience.

Please give it a shot and let us know!

Really great article. I've walked the same path and thought the same thoughts, and always felt somehow that it was just me. Here's hoping that Copilot lowers the barrier for all developers - especially ones who just have a small site or hobby project.

having tried everything from convox to fargatecli, deployfish ,etc. in the end i just ended up using pulumi to set up the basic infrastructure components and writing my own bash scripts to deploy tasks and services to ECS using the AWS cli.

there was way too much magic going on and i wasn't confident i could easily roll back or do modifications if the CLI didn't like something in particular. deployments based on terraform or cloudformation were particular susceptible to this.

This is pretty cool - very close to https://shipula.userbugreport.com/ that I've been cooking up to make my own life easier.

The example yaml file contains cpu: 1024, but the text says deploy on 1 cpu?

In ECS land, a single vCPU is described as 1024 "CPU units" that can then be distributed more granularly. This would allow you, for example, to evenly distribute a single CPU across 4 processes.

Reply on news.ycombinator.com