Architect – A Serverless Framework
blog.reifyworks.com
blog.reifyworks.com
If you like JSON and YAML and don't mind those issues you are well served by existing solutions. Cheers!
- JSON has no comments - YAML nesting tends to get deep which makes creating bugs very easy - Both are become hard to read / edit in larger files
The primary issue is translating the intent into a machine syntax. The format we landed focused on minimalism to communicate intent.
The the file format is the least interesting thing about .arc but one of the most distinctive. An .arc file VERY QUICKLY communicates the project intent.
An example:
``` @app hi
@html get / ```
The equiv CloudFormation YAML is 700 lines. You'd have no idea what that project does without digging into the code. The .arc by contrast is extremely easy to both read and author. As the app grows it becomes very clear the project intent and were to find things.
We certainly could have created a new YAML thing but many projects already do that … and I'll let their work speak for itself.
Accepted, but that's not the topic of debate.
The issue here is that you can write the example you gave in YAML like this
app:
hi
html:
get /
Hope I explained my point correctly.
This is just a matter of UX/ API, not how it works.
If you somehow felt that I disrespected your work, forgive me.
https://serverless.com/framework/docs/providers/aws/guide/se...
Obviously that's a verbose example and there are defaults for most everything AFAIK.
Anyway, Architect seems cool too! Always down for more options and competition!
YAML's shortcomings can be addressed a number of ways. The most recent trends I've seen are:
1. Simplify the YAML itself.
This usually means introducing higher-order features that make long, repetitive blocks irrelevant. An example is BOSH v2, which introduced a number of features (Eg. BOSH links) that pushed repetitive scutwork into the tool, away from the YAML manifests.
A variant of this is to use more advanced YAMLisms to make your YAML files more concise. Anchors make a big difference. I've seen this be very useful in large Concourse pipelines.
2. Use something that renders down to YAML.
For example, Jsonnet[0] or Enaml[1]. Some folks will use templating languages for this purpose (erb, mustache etc).
Templating is a last resort, in my view, because it's hard to see what the output will be without rendering it. Plus most tools format nested syntaxes poorly.
On the upside, use of a full templating engine introduces opportunities for multi-file DRYing. But that itself should be a smell that the configured tool is missing some feature or capability.
3. Use a different format.
TOML appears to be the most popular alternative to YAML and JSON. Most recently I have seen the joint Heroku-Pivotal effort on Buildpacks v3 adopting TOML over YAML or JSON.
Is it inherently better in a way I'm not seeing?
Would arc be suitable enough for a new project to adopt?
Its absolutely suitable today. We have it in production in several apps. begin.com and arc.codes itself are two examples that have seen non-trivial traffic.
The ability to have the infrastructure isolated from the raw code being written. (Think more of a grid computing and automatic planning being included)
Unfortunately, I'm not seeing it. I'm seeing that it's heavy on the initial business idea: "Get people to pay more and utilize our servers better".
Can you elaborate? It sounds like you're saying that if you write code that's deployed to a serverless infra you end up with 'serverless code'. This hasn't been my (very limited) experience - where the serverless stuff is just a thin layer around your business logic.
This neatly cleaves responsibility: your business logic team can just build+deploy+expose functions; and your frontend and ecosystem teams can build+deploy+expose the APIs they want on top, perhaps being maintained within their own codebases.
These can even be done on two entirely-separate platforms, rather than both being a part of the same “serverless” IaaS. You could use any combination of commodity FaaS providers (including your own hosted FaaS clusters) to deploy functions to; and then you could use some “smart” load-balancing system (e.g. Cloudflare Workers, or the routing component of an FaaS) to do the routing+aggregation on.
What I've seen AWS Lambda being used: One-off functions with manual output management. They aren't really being chained together.
I like some things about Architect, but there are things that just don't make sense:
- Integrations with Slack but no way to define custom CFN resources
- No authentication support
- Custom file format (3)
- Not being able to set CORS in this custom file format (4)
- Magical vendor folders (5)
I am sorry for the negativity, but brianleroux has posted in this thread several times claiming his framework saves 700-line CloudFormation templates. What I don't see is him comparing snippets with say, the Serverless framework(1), or AWS SAM(2). He seems to portray the choices as either his pet-framework-of-the-month or plain old CloudFormation, and that's just not true. That's not to say Serverless or AWS SAM are perfect (they most certainly have their warts), but they are both arguably more mature and have larger communities behind them.
(2) https://github.com/awslabs/serverless-application-model
(3) https://arc.codes/quickstart/arc-project-layout
No auth support is false. We support session out of the box and you're free to impl any auth scheme of your choosing.
Custom file format is being well addressed in other comments. If you don't like it thats fine: enjoy YAML elsewhere.
CORS could be implemented in the file format…I'm personally not a fan for a variety of reasons but its open source. PR's welcome.
Magic folders! Absolutely. Structure is a good thing.
I don't find this negative. Serverless framework is an abstraction that is easily as verbose as CF but I'm not interested in comparing to it because we have different goals. Serverless is an abstraction between clouds. If mulitcloud is your goal: use Serverless. SAM is a flavor of CF which is def less verbose but still very vendor specific and …wordy.
This isn't a 'pet framework of the month'. I do take mild offense to that. Development of .arc started in 2015 for begin.com and got met with some serious production traffic (from things like Techcrunch and Venturebeat). It held up beautifully. We decided to pivot from our original producton into CI/CD for arc with begin.com because it is clear to us that it is a better way to build serverless apps.
Architect was officially open sourced in July of 2017 and donated to the JS Foundation to demonstrate the long term commitment to it. Development has been very active since. I stand behind my work, dedication to open governance and approach we're taking and, I think, I have the track record to ask for the benefit of the doubt when I say that.
> No integration w CF is absolutely true but you're also totally free to do that.
It just seems then that there is no real way for me as a developer to transfer all the accumulated knowledge I already have about AWS to your framework. Looking through Architect's source, it actually seems as if everything is done via the AWS Javascript SDK, which seems a bit ... odd, to say the least. I'd even venture as far as saying it goes against the AWS Well Architected Framework guidelines (1) (which recommends using CloudFormation).
> No auth support is false. We support session out of the box and you're free to impl any auth scheme of your choosing.
Apologies, I couldn't find any mention in the docs. I meant more in terms of using an AWS service such as Cognito Federated Identities, and in terms of protecting API routes with either AWS_IAM, API keys or Cognito Authorizers.
> CORS could be implemented in the file format....
An Architect project deployment is not exactly repeatable if I still have to go to the Web Console to enable CORS now, is it?
> Magic folders!
I personally would prefer either my project's root `node_modules` or a Webpack bundle be deployed, over which I have some measure of control. Node already has a hard enough time with its resolution algorithms, must we add to the madness?
> Serverless framework is an abstraction that is easily as verbose as CF
I can't agree with you on Serverless being as verbose as CloudFormation, and I think you should know better as someone who has dealt with the complexity of setting up API Gateway programatically.
> but I'm not interested in comparing to it because we have different goals.
I think the comparisons are fair - both Architect and Serverless claim to be "Serverless frameworks". I could be in the minority though.
> Serverless is an abstraction between clouds. If mulitcloud is your goal: use Serverless.
I mean... you're not wrong, but that's grossly oversimplifying it. I've personally added support for Serverless AWS projects to allow hooking up Lambdas to Cognito User Pool triggers. That's a pretty solid AWS-only use-case, that likely isn't available on other cloud platforms (or at least, I never implemented it for them).
> SAM is a flavor of CF which is def less verbose but still very vendor specific and …wordy.
Well, isn't Architect vendor-specific as well?
> This isn't a 'pet framework of the month'. I do take mild offense to that.
Apologies for that remark. I'll be honest - I'm not really interested in the story behind the framework, I'm more interested in what it brings to the table. I'm glad your project worked for you and your company, and I wish it the best, but personally I don't see why Architect couldn't have just been a layer of syntax sugar over SAM/Serverless framework. If there was a killer feature that's specific to Architect, I can't see it (unless you count the file format as a feature).
(1) https://d1.awsstatic.com/whitepapers/architecture/AWS_Well-A...
Serverless framework is alright, but I felt it was really bloaty. There was just so much going on. Too many variables and it confuses the user. This looks clean. I might give it a try. I didn't find a link though in your blog.
I've wanted to try building something as a serverless architecture for a while, but every time I look into it I get confused by all the different things you have to set up to get an AWS Lambda function working, or I look into a framework but I can only find pre-canned examples for hooking into Slack, or just generally get confused by blog posts with lots of buzzwords and very little substance. In the end I always wind up just writing another daemon and sticking it on my server to get the job done. All I want to do is write some code and trigger it with an HTTP request, but for some reason Architect is the first framework I've seen that puts an example for how to do that on the homepage.
If you want to get started building serverless functions and APIs quickly, do it in your browser with https://code.xyz - we just released it along with an announcement of our most recent $2M in funding from Stripe [1] who join Slack and others as investors and customers.
Disclaimer: I’m the founder.
code.xyz gives some 'hello world' examples, but it doesn't even link to any docs--it just goes "look how easy it is! Here, sign in to our service!" I went hunting for the stdlib docs, and it took me five minutes to figure out how to call my service by HTTP--apparently I just get a special username.stdlib.io endpoint that I can call, but when I clicked 'Calling Services' the first thing I got was a page that explained how to use bearer tokens. I still don't know how I can POST to this endpoint, but I have learned that apparently stdlib has invented the concept of a 'source code' because apparently having a pre-canned Slack bot integration is super-important to any self-respecting FaaS provider. Also apparently I can find prebuilt 'source codes' in the 'stdlib registry', but the docs don't link to said registry, only the aforementioned Slack integration. At this point I gave up; I don't have the patience to learn a brand-new meaning for 'source codes' when all I want to do is make an API call when I get a webhook payload, which I already know how to do in a dozen lines of Python.
I realize this complaint sounds kind of harsh; you're getting a lot of my frustration dumped on you. Really all I want is basically a managed, autoscaled version of Express.js. Maybe there's some reason it's like this (I've never worked with anything that could be remotely described as 'at scale', it's mostly been line-of-business apps with a few dozen users maximum, or at most a web shop that still runs on a single dedicated server), but it seems like all of the Serverless providers add a bunch of extra concepts and confusion on top of what should really be a simple principle. Or maybe I'm completely misunderstanding the point of Serverless and it's actually intended to solve a completely different problem I've never had--I'm also open to that explanation instead!
No worries re: frustration. It happens!
Let's say you have to define a bunch of services, with mostly the same environment variables. You can make a "defaults" node, and then a) reference that node in the service descriptions to re-use those env vars, and b) selectively override those defaults in the service description's node while leaving the rest of the defaults untouched. It's really nice.
This is useful, say, if you're actually de/serializing objects and want to maintain actual references, or if you're just trying to share data between nodes.
But then our db on AWS is another $110, where it's included in heroku.
I know it will depend on your memory usage, etc. and as we grow, we expect to move to ec2, but I'm curious as to where/how you had such a huge difference in cost.
It has been a month since then and the feedback was awesome.
I would love to hear your opinion and how you could possibly use it (use case), since you can do a lot with serverless/FaaS etc.
cheers
YAML is good, but not to everyone's taste. I've been using it for a long time (as a ruby developer) but many people I know find it ugly.
I think Arc config looks much better than anything they could have come up with using JSON/YAML. Optimize configuration files for humans, not machines. If its simple enough to learn and be readable, more power to em.
https://github.com/toml-lang/toml
Its the thing Pipfile uses.
The point is, don't invent file types!
@this-section-has-a-map
hello-world
name child-map
ಠ-ಠ a
ಠ_ಠ b
Only one of each section is what we designed for though the validations for that might be broken (and if so we can fix right away). Nesting is limited to one level. No need for more that we've seen yet.
edit--Especially when it comes to users editing config files. Imagine a new user trying to edit a web server/db/.bashrc file that didn't have comments to help out?
It aids in troubleshooting greatly to have inline comments point out "odd" settings in configuration without having to look at seperate (usually old) documentation.
Also, json has json schema which tools like VS Code support. They can autocomplete your config. Pretty promising over most other tooling.
But coming up with your own config has pretty much all the downsides. Yeah, you can invent some cuter syntax. But now your users can't even get syntax highlighting in their editor. Awful UX.
Seen plenty of JSON without quoted keys before that just seemed to work, so depends on parser.
Albeit, I'm not fully sold on this whole fake-serverless architecture.
A major part of it is that AWS is arguably frustrating, since the docs are __clearly__ written by marketing people, not engineers. Every aspect feels non-dev friendly.
I would love to see something like Captain Duck Duck, so you can host your own serverless platfrom, sort-of a personalized AWS?
(Captain Duck Duck is a personalized Heroku)
But this is the killer Node and NPM framework...
You are pulling a straw man by implying that requires much cloudformation, which is completely unrelated. Besides, CF is a good example of how you can use two different syntax to describe the same information.
if so, leading with "just not true" contributed to this (which seems like a non-sequitur if you're not saying the claim is false)
Every time a function runs, it has to create a new connection to the Postgres database so it can run a query. This is super slow, and dead in the water. If I have to maintain a separate server to have a global connection pool, I might as well just have a server for my API as well.
What does "serverless" framework really tell veterans? Who are you selling to? What are the considerations for adoption?
There's so many questions that pop up before I even look at the software, and once I did, there's more questions which obscure the point. Why should I use your bullshit?
Worse, 5 years in after a company has adopted this junk, some poor sap is going to have to maintain it after being 100k SLOC deep in it, with no vision of how to port out of it alongside meeting existing technical task completions, at which point this will have become completely irrelevant.
To me, serverless is as normal a progression of abstraction as anything in software development. What about FaaS crosses a line for you, where (presumably) PaaS doesn't?