Libaws: A simpler way to declare AWS infrastructure
github.com
github.com
is Linux easy? Is Windows?
AWS (as is GCP and Azure) can basically be thought of as an entire operating system, with extremely low level concepts (VPCs, EC2 machines), mid-level (Lambda, StepFunctions, EventBridge), and high-level (Translate, Comprehend, Rekognition, etc)
I am not sure if this library helps. It presents its own opinionated flavour of AWS but doesn't really hide enough of the details.
If you want to use AWS and not think of underlying infra like networks and security groups, build on high-level services like EventBridge and StepFunctions.
Alternatively, if you don't want to think about services at all, use CDK constructs: https://docs.aws.amazon.com/cdk/v2/guide/constructs.html
For example, ehre is a NetworkLoadBalancedFargateService: https://docs.aws.amazon.com/cdk/api/v1/docs/@aws-cdk_aws-ecs...
There exists a cohort of internal customers at medium-to-large sized software companies that need to ship new applications without being bogged down by the decisions of how to get that thing running and playing nicely with the rest of the ecosystem. For the same reason they don't give their end users CRUD APIs to use their products, it doesn't always make sense to give our developers the entire suite of AWS and say "figure it out".
There's need here but I'm not sure we've figured out what the right answer is.
The problems I’ve seen is that like most software in general there is no one size fits all approach to scaffolding and management infrastructure in AWS or any other cloud, especially when we add in the realities of business pivoting being likely.
On the other hand, there are offerings such as AWS Landing Zones that sorta help if you’re running a big ol’ sprawling company but it’s totally overkill and costs more than anyone really wants to pay for if they’re just getting started.
IMO, the AWS journey is probably better off for most folks using Lightsail and scaling with several of those instances making sure they can be containerized and have a brain dead easy deployment workflow that can withstand instance failures before getting more complicated. A lot of people are pushing serverless at first but it’s cumbersome enough for application developers that it’s distracting from the most important thing early on - building your dang product, silly.
Otherwise I’d suggest going hard into containers and using a PaaS until it really, really hurts to where you can afford to pay the eye watering compensation of most decent SREs (expect $15k+ USD / month total spend if you have few advantages in terms of access to the market).
I am not a fan of over-planning infrastructure at all but have also seen what happens when developers take too long to hire someone specialized in infrastructure and has seen the common mistakes that can gut company growth hard as they’re trying to scale up and out.
Well, if it's a medium-to-large sized company they will be bogged down by governance processes anyway. Who approves the budget for this? Who signs off the cybersecurity and data protection compliance policies?
The classic example is public-read S3 buckets. My favorite-er example is actually a security guidance which reads: nothing should use the default VPC (usually phrased from the perspective of, all the SGs on the default VPC should restrict all traffic). Follow that line of logic: why did you give me a default VPC then? Well, they'd argue, it makes onboarding easier, you can just deploy all these cool resources without worrying about the complexities of the networking. Wait hold on. All of this is within their power to control, they made the networking insanely complex, they made the networking layer integral to the Cool Product, they created the idea of a default VPC, they created the security rule that says its bad, and it really paints this picture of a house of cards where no-one is in control, no one has vision, its just a bunch of heads arguing with each other about their worldview on how things should be.
Of course, one head yells "backwards compatibility, legacy, more knobs more knobs" and because its Amazon (and Microsoft) that head gets a megaphone.
Because of this, as the OP wishes, AWS accounts are an extraordinarily useless thing to hand over to a development team. They're nothing like a Heroku or DigitalOcean account; at best, they're so complex that you need a domain expert to understand what to do, so all that ends up living in the CloudOps Org. At worst, the team will shoot themselves in the foot, and maybe take the company's bank account (or worse, customer data) with them.
Thus, the New Hotness in BigBoyTech is to build a PaaS on top of AWS. Because Amazon isn't capable of doing it! Lightsail is a joke. Lambda is probably the closest they've gotten. This PaaS asserts the security, compliance, performance, architecture, etc opinions the company wants, and dev teams can build on it.
Fine, whatever, but at some point you have to question why we're so obsessed with inefficiency that we tolerate knobs we won't use, or we'll miss turning to Secure, when alternatives do exist. Do you know what a compliance official's wet dream is? "Hey man I need somewhere to store this file in the cloud" "Use Google Drive" DONE! They bought Google Drive. Maybe tweaked a couple settings. Secure by default. AWS is nothing like that, and it can take millions in engineering to get an active AWS account to that state (let alone delegated management of multiple accounts across an org).
It's actually collective insanity that we put up with it, when the only half-reason why it is so is because JP Morgan has a COBOL service built in 1987 that can only run if its hardwired with a telephone jack directly to an ISP interchange in Ashburn, and they're too cheap to modernize it so just tax the rest of the world by abusing Amazon's "never say No to a paying customer" policy. I'm being dramatic, but that describes 98% of the knobs on AWS.
Everything built in AWS was done with Terraform (minus some bootstrapping of accounts and orgs). Custom terraform modules were built which were guardrails of underlying raw Terraform providers.
I'm not saying it's ideal, I'm just throwing it out there.
"Piloting a passenger plane should be easy"
"Performing brain surgery should be easy"
None of that should be easy. AWS is not meant to be focused on ease of use in the first place. It offers brutal power to those who know how to operate it and brutal ways to fuck up if you don’t.
The shipping of applications mentioned by the comment in this thread is not a valid statement either.. there are countless services by now attempting exactly this (Firebase e.g), yet, demanding that AWS or GCP should be child's play to use shows a complete disregard for what they actually offer.. I feel like this is similar to saying "we want to build a spacecraft to transport satellites into space, but we'd love to just have two buttons to start and land the rocket"
And then, even if you manage to abstract everything away to just those two buttons, someone will press it in the wrong moment and the rocket hits a plane flying above and you realize the expertise required is always required and someone who has this expertise doesn't need all of the abstraction in the first place.
> "Performing brain surgery should be easy"
> None of that should be easy.
Why shouldn't those things be easy? Wouldn't it be better if they were? It would mean more people could do it well (I'm not saying let anyone do it!) and it would presumably have come about as a result of better understanding and tooling. Imagine if people didn't die on the waitlist for the best neurosurgeon in the country, because we'd made brain surgery easier and so now all the neurosurgeons could do that one formerly-super-difficult surgery.
Medical care is sometimes extremely time sensitive and you don't want the doc to have to ask his superior how to stop your bleeding.
On the contrary, I think it would be a lot easier to treat an outlier condition because doctors would have a lot more time to expand their skills and study uncommon conditions.
Why would that necessarily be the case? Why wouldn't we train doctors to still know what to do, and also make it a lot easier to do it well?
Not complaining thought, I make good money understanding AWS so others don't need to.
> Why should it be?
Because if they don't focus on developer experience, they might end up being treated as a commodity. Eventually someone will eat their lunch.
There is no one AWS philosophy. The philosophy of foundation networking services is going to be different from higher-abstraction services.
They are for different customers with different needs.
> Because if they don't focus on developer experience, they might end up being treated as a commodity. Eventually someone will eat their lunch.
That's certainly true.
This is the level that it makes sense to use AWS at if you're a small team. 10 lines of code/5 minutes and you have a deployment of https terminated load balanced containers that scale with your usage. It just works, it scales, and for small teams it saves a huge amount of time (and Dev effort - instead of focusing on setting up the correct nginx or haproxy settings I can write my app).
Linux is easy/well-designed/well-documented. Windows is a hackjob mess. There's a difference.
If you allow me a story, I sail on a kind of boat, called a squib. For the foresail, eg the sail that is in front of the mast, it has the regular sheets, a Cunningham, a halyard tensioner and barber haulers. The squib is a very easy boat to sail, notoriously hard to sail fast.
Complaining about AWS being complicated, is the same as complaining the squib is complicated, just because it gives you options. I don't have to use all those controls, I still can get from a to b without using them. But if I want to win the race, I have to understand what they do and use them appropriately.
> proposes solution that involves YAML
Am I crazy, ignorant, or is YAML the most tedious and error-prone format edit?
I lile python, I like significant whitespace, I hate how YAML does it.
Python, Go, even Scratch (for a 100% GUX)… And among the many pros, error handling is enough to warrant the change from markup.
But IMHO here's the one barrier: people need to stop thinking of an Information System (IS) as a collection of machines, and move one level of abstraction above to consider said IS as the unit, and machines as subs of sort (subsystems, components, modules, call it what you want). Conceptually, your "main" space is the IS, everything else is modules down the namespace hierarchy. We need to think of machines as we used to think of programs, and think of IS's as we used to think of machines. In the UNIX philosophy, any application is a collection of programs, just as an IS-as-code is a collection of machines-as-software-modules (basically feature libraries for "main" to use).
When that paradigm comes, you'll be able to simply "import" some subsystem (say Elastic Search, whatever machines in the IS) and add that (as a class, method, whatever) to all your existing objects, because the concept of "interfaces" (quite literally the basis of interoperability, dating back forever in computing) will be _native_ to your IS model in a Turing-complete paradigm.
Not sure my wording made sense to non-programmers (cue: "quotes" are technical terms, not general meaning), but I'm too lazy to write a layman expose.
The problem is more with the software that uses it, and it using YAML is just a high quality proxy for the signal you want.
YAML is a bit error prone, but it's those tools that have an incredibly complex format.
Projects like Pulumi and CDK seem to be much better approaches, but don't seem to have much traction compared to TF, Cloudformation, etc.
Then some tools came out which let you use YAML or JSON to describe the desired state of the world, and some tool would diff that against the current state of the world to determine which resources to create, which to delete, and which to update. People started to conflate "YAML" with this sort of diffing tool and they conflated imperative programming languages with the legacy imperative scripting approach. Early infra-as-code vendors capitalized on the YAML=good/programs=bad myth and re-emphasized it by showing cute toy examples and passing off the simplicity as an effect of the YAML/etc rather than the inherent simplicity of the example.
Unfortunately, pure YAML/HCL/etc doesn't scale--you end up needing to reference resources from other resources (and/or attributes on those resources), and you end up needing to DRY up many repetitious blocks and so on. What you naturally want is to generate those static YAML configs from a programming language; however, IaC vendors had committed themselves to the "YAML = simple" brand so instead they started building half-assed programming language features into their YAML dialect (effectively using YAML to represent abstract syntax trees for the world's crappiest programming languages). Terraform and CloudFormation fall into this bucket.
I guess Helm looked at the landscape and decided it wasn't easy enough to generate syntactically invalid YAML blobs, and consequently decided to use text templates.
Eventually the industry caught on to the scam and demanded programming languages back, so we got CDKs which misunderstand the assignment in a different way. Rather than emitting YAML/etc that can be passed into a diff engine, you get some weird bindings that calls (and is called from) a node process (I can't tell what this node process actually does even after reading the docs).
Troposphere emits YAML.
You can diff the outputs if you wish, or did I misunderstand your last paragraph.
These are two examples I found online. The first is more complicated but it doesn't do any JavaScript IPC, no inheritance, no mutation, etc. You write it just like you want to write YAML, and it straightforwardly emits YAML (rather than the CDK version which is more opaque/magical). I prefer this:
def main():
bucket_name_parameter = ParameterString(Description="The name of the bucket")
key_arn_parameter = ParameterString(
Description="The ARN of the KMS key used to encrypt the bucket",
)
bucket = Bucket(BucketName=bucket_name_parameter)
t = Template(
description="S3 Bucket Template",
parameters={
"BucketName": bucket_name_parameter,
"KMSKeyARN": key_arn_parameter,
},
resources={
"Bucket": bucket,
"BucketPolicy": ManagedPolicy(
PolicyDocument={
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowFullAccessToBucket",
"Action": "s3:*",
"Effect": "Allow",
"Resource": Sub(
f"${{BucketARN}}/*", BucketARN=bucket.GetArn()
),
},
{
"Sid": "AllowUseOfTheKey",
"Effect": "Allow",
"Action": [
"kms:Encrypt",
"kms:Decrypt",
"kms:ReEncrypt*",
"kms:GenerateDataKey*",
"kms:DescribeKey",
],
"Resource": key_arn_parameter,
},
{
"Sid": "AllowAttachmentOfPersistentResources",
"Effect": "Allow",
"Action": [
"kms:CreateGrant",
"kms:ListGrants",
"kms:RevokeGrant",
],
"Resource": key_arn_parameter,
"Condition": {"Bool": {"kms:GrantIsForAWSResource": True}},
},
],
},
),
},
)
print(json.dumps(t.template_to_cloudformation(), indent=4))
class S3Stack(Stack):
def __init__(self, app: App, id: str) -> None:
super().__init__(app, id)
self.access_point = f"arn:aws:s3:{Aws.REGION}:{Aws.ACCOUNT_ID}:accesspoint/" \
f"{S3_ACCESS_POINT_NAME}"
# Set up a bucket
bucket = s3.Bucket(
self,
"example-bucket",
access_control=s3.BucketAccessControl.BUCKET_OWNER_FULL_CONTROL,
encryption=s3.BucketEncryption.S3_MANAGED,
block_public_access=s3.BlockPublicAccess.BLOCK_ALL,
)
# Delegating access control to access points
# https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-points-policies.html
bucket.add_to_resource_policy(
iam.PolicyStatement(
actions=["*"],
principals=[iam.AnyPrincipal()],
resources=[
bucket.bucket_arn,
bucket.arn_for_objects('*')
],
conditions={
"StringEquals":
{
"s3:DataAccessPointAccount": f"{Aws.ACCOUNT_ID}"
}
}
),
)
You’re in for a shock when you realise all they do is emit YAML
I think it's a pleasant middle ground between config and code, it's definitely code, but limited.
But it can be tricky to read other people's YAML files on GitHub. It's easy to miss subtle mistakes.
Though better than when Jinja is used on YAML.
Do you edit Python Pickle files by hand?
PS: I forgot that I wrote a cloud init file merely hours ago. Yuck.
(Yes there is a comparison to 'terraform' in the Readme - I'm not convinced it's considering modules, versus just using the AWS provider directly. Another bonus of using tf modules over this: you can eventually realise you need to graduate to using the AWS provider directly, and do so quite easily (cf. `terraform state mv`).)
The AWS CDK has higher level "constructs"[1] that are really nice to use.
How does that compare to the state of the world for things like Pulumi or Terraform these days?
0: https://github.com/lunasec-io/lunasec/tree/master/lunatrace/...
Any of these files represents the infrastructure. The "bin" folder is the actual entry point.
1: https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_...
We're using this as well as the Fargate SQS worker one to scale up backend job processing. Lambda is a pain and this works quite magically.
https://github.com/cfn-modules/docs
I would personally switch to CDK, but of you wanted to add predefined templates you don't need to switch your existing languages.
AWS provides extreme flexibility. This really only becomes valuable once you hit a scale that you can optimize your infrastructure around the shape of your needs.
* Need some heavy workload to run in the most cost effective data center at the most cost effective time. Yep, AWS is great for that.
* Need a specialized GPU set. Yep, AWS is great for that.
* Need some crazy storage or memory setup. Yep, AWS is great for that.
For everything else, you should really consider using a PaaS with a good ol' Docker container.
This is upside down? When you hit the scale where you get value from modelling your infra around your business use cases that's when you go on-prem or use services that let you rent bare metal. Before that you use cloud.
You get something working in a docker, push to your AWS repo, and App Runner grabs it and goes.
Maybe I'm interpreting this wrong and the HN-consensus path is something like bare VPS => Docker => use AWS directly and tweak everything how you want it?
For the most part, the constructs provided out of the box has good defaults and the documentation helps. The aws-cdk support team is also responsive on issues/bugs reported.
This is just based on my experience and I started off as a total noob on AWS.
The hard part isn't making AWS easier, it is making a tool which is both easy and flexible enough that a user can do what they need (within reason), then outgrow your tool and move on. In specific any tool based abstraction has the downside that if the only way to interact with your infrastructure is through the tool then you severely limit the options for a user to make customizations. AWS will add features faster than you can keep up, and there will always be someone who can't use your tool unless you have feature X, Y, and Z.
At that point you can either add more and more features until your tool becomes as complex to use as all of AWS itself, or you can draw a line in the sand and say "you have outgrown my tool, it is time to move on". When they reach that point you should try to make it easy for them to avoid having to recreate everything from scratch themselves.
For AWS Copilot we tackled this problem in part by adding a "copilot svc package" command which spits out a full CloudFormation stack for your application and all its infrastructure, so that you don't have to restart from scratch if you decide you have outgrown Copilot. Copilot will help you deploy simple Docker based container applications quickly and easily, but then if you have more complex needs you can "graduate" on to the next experience by exporting things up into a stable, productionized CloudFormation template that lets you customize the full range of AWS settings and resources to your hearts content.
I think your tooling will need something similar. The "outgrowing" section (https://github.com/nathants/libaws#outgrowing) is a bit weak right now, and that's what you need to really nail down for people to have the trust to start adopting a tool for serious usage. They need to know that once they outgrow the tool they will have a viable path that won't be a huge headache.
Otherwise I like your ideas here, and I think the YAML based DSL here is a fascinating abstraction of many common, simple use cases. Great job!
Cloudformation or AWS CDK are upgraded slower than providers for example.
So this argument is mot.
I'm specifically speaking about tooling that attempts to abstract and simplify the AWS experience. This is where the trickiest decisions have to be made about which properties and options to surface, how to surface them if so, and which ones should just be left out and require the user to drop down a level to the lower level provider as a fallback.
Managers are long looking for a way to hire cheap labor. Issue us cheap labor lacks knowledge required to build soecialized things. And thankfully it will never change, no tool can provide that.
Sure, but vanilla setups should be trivial and optimised for cost. That is not to their or their consultants’ benefit.
I mean, whenever I asked someone ‘how do I do this in aws’, they tell me ‘oh that is a bog standard blah setup, very Simple’. So if that is simple, why is it so hard to set up as a novice (with 20 years of rack server bare metal hosting experience). I had to learn everything about aws in order to figure out that all experts saying things are ‘simple’ were still not providing optimal solutions for my vanille, and for decades unchanged across many different applications, hosting wishes. If you don’t know literally everything about aws and hired a consultant or a team to set it up, you are probably (massively) overpaying. And often, as I see at clients, for vanilla stuff that should be 1 click setup. But yes, I guess tool providers should provide this on top, not aws itself.
It really needs some love from a higher level thinker who's not steeped in the solution, but in the problem. Presumably this is a PM but there's just not a lot of good ones in the org to be frank. They're almost not worth consulting.
AWS is a low-level tool, and that's okay.
from aws_cdk import aws_s3 as s3,naws_iam as iam,bcore
class S3Template(core.Stack):
def __init__(self, app: core.App, id: str, **kwargs) -> None:
super().__init__(app, id)
myBucket = s3.Bucket(self, 'MyFirstBucket',
bucket_name='balkaran-aws-cdk-s3-demo-bucket')
myBucket.add_to_resource_policy(
iam.PolicyStatement(
actions=['s3:GetObject'],
resources=
[myBucket.arn_for_objects('*')],
principals=
[iam.AccountPrincipal(account_id=core.Aws.ACCOUNT_ID)]
)
)
myUser = iam.User(self,'s3User')
myBucket.grant_write(myUser)
app = core.App()
S3Template(app, "S3Template", env={'region':'us-east-1'})
app.synth()I do sympathize with the idea though, as I was recently trying to deploy a container for a small project, and didn't want to deal with all the complexities a fully featured cloud provider has. I found AWS Lightsail, GCP Cloud Run, and later settled on fly.io. It's hard for me to imagine using this tool from the beginning, or even graduating to this tool instead of upgrading to plain AWS and maybe Terraform.
Is this IaC "light" for purposes of setting up and tearing down small applications for experiments or demos?
Obviously a lot of effort spent. Of course, once you know AWS well enough to "make it easy", by that time, you've found the need for tooling which supports the harder stuff. And, frankly, someone new to AWS needs a lot of context just to understand the canonical ways to wire stuff together in AWS to even comprehend which pattern will be useful.
For example, S3 events are really backed by SQS under the covers, with implied retries on failure, but no DLQ support, and a lambda that triggers on an arriving S3 event may need to account for "duplicate" events, etc. That's just one wrinkle that even a simplified configuration might require more complex understanding of the underlying AWS ecosystem.
This project seems to be a collection of pre-assembled lego parts, that can eventually be put together in the form of a robot monkey. But I think all of us actually just want the robot monkey, not several clumps of legos. We should have more feature-complete projects that we can push a button and it pops up for you. But that's very different than saying AWS should be easy.
I think we should stop obsessing over "code", and start focusing on robot monkeys. In other words, stop thinking you need more and more and more abstractions around calling AWS APIs, and focus more on publishing a complete working solution for a specific problem.
I don't give a shit about SQS queues and CloudWatch alerts and Lambdas and IAM role policies! I give a shit about a file-upload-website that automatically triggers a python program to process a file and deliver the result into an S3 bucket or Postgres database. That complete thing is the robot monkey. That's what we should be publishing; not a million lines of boilerplate that I still have to spend time and energy cobbling together into a robot monkey. I have built the same robot monkey over and over and over again in 5 different abstractions. I don't care what the abstraction is, I just want the robot monkey!
I think the reason we don't have more robot monkeys is that they're built within companies out of "code", and so we aren't allowed to open-source them, because it's "intellectual property". We need a new abstraction that is not code, and is not considered super secret proprietary information, so we can publish it as open source without thinking.
For example: what if you could point a tool at some part of your AWS account, and it exports a declarative configuration file that is a snapshot of how everything is plumbed together? The tool could strip out any "literals" and anonymize all strings, so that there is nothing proprietary in the end result except a lot of API calls to functions with random names. The end result would essentially be an unlabeled architectural diagram that you could execute. You could then publish that on the Robot Monkey Database with a description of what it does.
This makes a pretty sound argument that “cloud-agnostic” solution is never going to be built.
Its bad for cloud-providers business and so they will never unify.