AWS Publishes Reference Architecture, Implementations for Deployment Pipelines
infoq.com
infoq.com
And a talk on what a cell based architecture is here: https://youtube.com/watch?v=HUwz8uko7HY
Cells are basically subsets or shards of customer resources with infrastructure isolated from other cells. The reference deployment pipeline in the blog post [0] would deploy to each cell individually in different waves depending on how your team configures the waves ultimately limiting the blast radius of a deployment to a smaller subset of customers.
[0]: https://aws.amazon.com/blogs/aws/new_deployment_pipelines_re...
[0]: https://buildpacks.io/docs/features/reproducibility/
[1]: https://nixos.org/
Or is there some other framework named "Brazil?"
Autocorrect gone viral?
I was thinking someone could probably build a manyrepo solution similar to Brazil on top of an idea like this. I don’t see many tools like Bazel built for manyrepos.
Edit: It's blog spam, sadly. Here are direct links to the actual blog posts and tools.
https://aws.amazon.com/blogs/aws/new_deployment_pipelines_re...
https://pipelines.devops.aws.dev/
https://github.com/aws-samples/aws-deployment-pipeline-refer...
https://aws.amazon.com/builders-library/automating-safe-hand...
Also, one-box/one-cell deployment stage before your production stage. Beta/gamma are well and good, but they’ll never perfectly replicate an actual prod deployment
Do companies run apps that get 1k rps on stuff like cell-based-architechture (mentioned in top comment)?
FWIW, I work at Amazon, and a lot of things are still solved with "Boring" tech that's not "web scale". Even without our own ranks we've got to remind people that engineering is about fitting the architecture to the requirements and no more. It seems to be lost knowledge these days that an RDBMS gets you really, really far. The vast majority of apps -- even those that appear superficially "large" -- will never see usage patterns that justify anything more than a box and a database. You'd be surprised which systems are just humming along unceremoniously with a humble 3-tier architecture
You can't build a Tier-1 service like S3 or EBS on top of RDS. If you need a database solution to help you with building and running tools of that scale, it has to be something else that is a lot less complex and a lot more robust.
It's okay for RDS to be dependent on a Tier-1 service like S3 or EBS, but not the reverse. You don't want to get into priority inversion problems here.
And when your service has to be in every AZ in every region, and they all have to be kept separately running, then things get even more complex.
Then you have to ask yourself what is the bootstrap process for building a new region.
For me the Deployment Pipeline starts with the Acceptance Tests that ensures business quality running over already created artifacts. There are advantages to this approach, as different deployment pipelines may be required to deploy to different platforms even if all artifacts are generated by one build pipeline.
AWS seems to mix the concepts of Build Pipeline with Deployment Pipeline.
Deployment has had historically a meaning in software development, I do not understand why to change it now.
>> Deployment has had historically a meaning in software development, I do not understand why to change it now.
Maybe because traditionally software shops would run a build server on premise, and a deployment on AWS. Now Amazon want's to move the build process on AWS as well - making them a little bit of extra money.
That may be the reason. To create an article that looks like professional best practices but it is just an advertisement. Microsoft used to do a lot of that in its war against Open Source back in the day.
Conceptually, it's easy enough to put these two sets of components together in the same "pipeline" tool. You just have to make sure that all the right steps and components are available, and that they are properly used. If so, you can set things up so that all your code goes through multiple online checks when it gets uploaded to the repo, and you prevent the code from being able to be merged to "live" (or "head" or "master" or whatever you want to call it), unless all the checks have passed.
Once all the checks have passed, your pipeline can proceed to push that to live (possible with a human approval required), then deploy that to beta, then gamma, then a canary "onebox" in one AZ in one region, and then finally into your first real multi-server production environment in one AZ in one region. Then you can chain on from there, with potential tests that have to be passed at each stage, and baking in periods that are required, and then on to the next stage with the next onebox in the next AZ in the next region.
Rinse and repeat.
Of course, you also need auto rollback processes, in case your deployments fail. And what happens if you need to rollback the rollbacks? Ad infinitum.
To that level, it's all just logical extensions of the initial concepts.
CDK is basically a make for makefiles using imperative code.
So you can just as easily target terraform with it.
You should also deploy new code to some kind of non-production environment before going to prod. Even companies with the best possible canarying in prod will still do that.
Some of the technology involved is definitely AWS-scale, but the principles of software rollout are broadly applicable.
Your deployment pipeline most likely looks like:
- Build your language artifacts
- Build a container
- Deploy that container somewhere
- Run approvals like integ tests, unit tests etc. and monitor metrics to decide whether to deploy to your next environment or rollback.
This is just a structured way of building these CD pipelines you could also use Jenkins or Google started building something called Skaffold for Google Cloud Deploy: https://skaffold.dev/docs/
As your company gets larger you'll want to limit the blast radius of new changes so that you can meet your SLAs and you'll want to use multiple AZs and cloud regions for HA. That's where the idea of waves and cells comes from.
for ordinary service use case or web site.. most probability do not need this. over engineering infrastructure when should focus on customer and problem domain..thing that increase top or bottom line.
source: principal SDE who launched several services at amazon using cell architecture.
That can help smaller teams become more agile, and being able to walk through a consistent process across multiple products for each step from commit against a dev branch through deployment in production makes it easier to account for team members being away, training new folks, etc. This is especially important for "side projects" or other small services that end up becoming critical infrastructure but are really maintained by one or two team members and only have a couple of internal customers.
That said, a base version of this pipeline is very useful even for a small startups. I concede that it takes a bit of time (took me about a week) to setup end to end. By end to end I mean beginning with empty AWS account to setting up VPC, ECS, CloudFront, Fargate, Codepipeline and so on. But once set you don't have to touch it for months on end. Scaling up/down, if needed, is as simple as minor config change.
[1] https://aws.amazon.com/blogs/aws/new_deployment_pipelines_re...
That having been said, the "reference architecture" being offered just feels like a slow day in the AWS marketing department during a recession. Nothing really special about it unless I've completely missed something?
At $dayjob, AWS consultants regularly "assist" with architectures where the final solution is invariably draped across every one of the AWS proprietary offerings. Whether on purpose or not, the result is that customers are locked in and the budgets are often a "surprise!".
[1] https://vercel.com/blog/behind-the-scenes-of-vercels-infrast... [2] https://twitter.com/cramforce/status/1619082343259705344