3factor apps: a pattern for fast iteration, resilience and high scalability | Hacker News Reader

3factor apps: a pattern for fast iteration, resilience and high scalability

3factor.app

So this pattern forces specific technology choices on you (GraphQL), and encourages vendor lock-in by going all-in on "Serverless" cloud functions. No thanks.

Well, you could also “serverless” with kubernetes and vendor + OSS ecosystem around running containers with 0-n scaling aka serverless is also getting better.

Also, GraphQL is important because it’s a convenient way for clients to consume asynchronous information.

OpenFaaS

We do our "serverless" using the self-hosted version of Azure Functions - Azure Webjobs. For most of the events we can scale out to 32 beefy machines. We also don't use graph SQL, preferring simple rest endpoints made in Web Api 2, and some reporting functions. If we need to do more than that...well, we'll have to rethink some things.

It will be a good problem to have.

While I agree Azure web jobs are awesome, I’m not sure I would call them serverless. They require you to provision an app service and would only scale within whatever tier you have and ultimately share resources with any sites you have on the same sever.

Microsoft actually has an on-prem or self-hosted version of Azure functions that uses docker, though it’s still only in preview.

After websockets got available in all major browsers I make almost all apps this way, but I use plain JSON instead of GraphQL. A request can look like this: {person: {name: "Jon"}}.

The way they pitch it, yes. In reality you could use any data store and any backend language/whatever and you would get the same affect. What they're pitching isn't really revolutionary - it's basically putting a queue between your mouse and state management/business logic.

Scaleability means more than just being able to accept infinite requests without buckling. It also means writing predictable code that your teammates (or you 6 months later) can parse and understand whats going to get touched. I would argue thats more important for 98% of the stuff we build. You can iterate faster when you can understand what your changes touch and see the whole picture. Talking to what is essentially a bunch of stored procs on AWS or similar and/or letting your front end team write the database queries to fetch data seems like a massive footgun. Am I missing something here?

Hm...not sure why you think this goes against what you’re saying. The idea is just to make things more asynchronous and give the front end team the ability to consume async information. Also, giving the frontend team the power to query what they want is not a bad thing. You should have a process to whitelist queries before they go into prod anyway.

Incidentally serverless can also mean microservices. The idea is just to deliberately remove state from inside the code and put it into state or events. And then running that on something that scales 0-n. Why Is that against your principle of scaling the codebase for the team? It’s just a dev principle to make ops easy.

This approach doesn't prevent doing things well - there's lots to like about eventing and async processing.

The distribution of the business logic among asynchronous actors (the serverless message-driven elements) will likely increase the potential for consistency issues at runtime and might increase the cognitive burden on the team at dev time ("where's the right place to put function X", "how do I test scenarios now that they're distributed", "what should my unit do if a collaborator didn't process its events in a timely fashion or at all").

These are all tractable problems, but problems nonetheless.

Let me start this by saying I am a frontend developer. Lest you think I am some curmudgeon coming at you with exclamations of the "good ol' days" of PHP/jQuery...

I'll give you this: you can build something suuuuuper fast with this architecture. I imagine it would be thrilling for the first, say, 6 months or so. Hell maybe even a year if your business rules don't change too radically. In SV that might well be enough. As long as you are building and not maintaining/updating I doubt you'll get bit too bad. Depends on the size of your team. In this case smaller would be better.

Its just that I've yet to see a compelling case for this type of thing for non FAANG companies. They have very very different problems than the rest of us and every one of them started out as a monolith (shoot Google STILL rocks one). To me it makes way more sense to start out as one, maybe two, code bases and then scale off things that need these performance boosts to serverless processes etc as they are needed. The reason being that to be effective in a given system, one needs to be able to understand the ramifications of changes as they are made. Its much easier to do that when you can grok the whole system vs talking to an isolated process.

Let me step through your comments/questions:

> Serverless can also mean micro services

Oh I am aware. Dear god I am I aware... I manage many of these in a given day. These are even worse honestly because they have their own dependency trees and keeping those up to date/compatible is a friggin' nightmare. Then of course there is configuration management. Then there is the case of having to maintain multiple release versions of an app and needing to apply patches to those micro services. Oh and what if you have a multi tenant environment and those are running different versions? Its just a mess. Thar be dragons dude.

> giving the frontend team the power to query what they want is not a bad thing

On this we disagree. There should be a discussion about this. A pattern to follow and a data contract. The UI should be as dumb as possible and only know about that contract. If that contract is not met, things should go red. I'd be willing to concede that something like a reporting interface or a complex filtering scenario would make sense. I just don't think it makes much sense to let anyone pull down anything they want wherever.

> You should have a process to whitelist queries before they go into prod anyway.

We agree on this. However, at the risk of sounding snarky, thats what an API contract is.

> And then running that on something that scales 0-n

Again, this is not a problem you need to solve right away. If/when it does arise, there are a lot of steps you can take that are simpler and easier to manage than a bazillion isolated processes that don't know about each other.

Am I saying this is a uniformly bad approach? No of course not. It might make perfect sense for your org. I am just saying that for most of the type of work I've done, the real scaling issues that I've encountered come from premature optimization and unnecessary obscurity.

Thanks. This is helpful!

One question on the API contract, giving frontend teams power thing: what are your thoughts on things like firebase on the one hand and graphql on the other?

I can see how you can his could be something to overcome or as something that works the same or better. Let's say a web front end does operation 'a' on service A, which mutates service B. Synchronous calls have issues we're familiar with. Async could mean that we subscribe to B and get notified when action 'a' is done. On the front end even an Ajax call is async so there's very little difference. For the backend you can get better reliability/resilience with async.

Looks like a multitier architecture plan to me. I see an app tier, a logic tier implemented in GraphQL, a state tier, and a functional services tier.

https://en.wikipedia.org/wiki/Multitier_architecture

Also see the 12 factor app guide, which aims to improve adaptability and resilience.

https://12factor.net/

I think this really boils down to:

  1) Have an api gateway (for them, graphql).
  2) Store state in a (most likely distributed) store.
  3) Have services interact with state via an event system.

All this really says is make your applications stateless. This is a trend that has been going on for a while.

I don't think GraphQL nor serverless is required and are just specific to their implementation.

Unless I totally missed what they were going for.

So basically you've renamed your API layer, graphQL

one note, using serverless doesn't allow you to scale to infinity. You are still dependent on your database/state management.

I don't have any experience, but isn't it generally encouraged to use DynamoDB or spanner for faas on aws or gcp because of that?

Its possible, but that that scale how your model your data, and what and where you want your data to be become very important.

If you need global state, then dynamo _might_ help, but that's still limited to a region. Yes there are mechanism for syncing regions, but you need to decide what's more important, consistence, or speed.

tl;dr:

>[An] architecture is analogous to the redux dataflow model on a react app, but applied to the fullstack.

The page has no concrete recommendations as for what tools could work to implement this pattern except GraphQL. Without a pretty strong knowledge of the tools out there, you won't know how to implement this architecture pattern well.

There's a lot of hand-waving past non-trivial problems and there's not much help on what tools/patterns/best practices will help that. For example, the architecture requires a fully async backend, yet that all state changes be atomic.

Similarly, the architecture requires `Your event system should deliver the events reliably to other microservices.`. Not a trivial thing to achieve.

Also, this architecture promises to `Scale your backend infinitely` because of using serverless services, but because everything is being funneled through a centralized state, That server/database has the potential to be a bottleneck that keeps you from scaling, and the architecture has no advice as to how to build your state to be scalable.

Hi HN!

Wrote this up based on our experiences working with early adopters of GraphQL and serverless. There have been lots of teething pains with serverless architectures and we've seen this as being a useful way of pushing business logic to being "serverless" which is run asynchronously and the results are captured asynchronously too. That way you don't have to worry about cold-starts and a lot of error handling and resiliency concerns go into the generic event system.

GraphQL subscriptions also fit in really well because they allow clients to consume asynchronous results.

Feedback, ideas, experiences very welcome!

Should list some concrete implementations. For example I think AWS AppSync with AWS Lambda is exactly this architecture as a service.

The issue I have with your website compared to the 12factor one is how hard it is to grasp/implement your advices.

> 12Factor Nbr 3 : Store config in ENV variables.

Yep I can do that. In 2 hours I can convert most of my conf in env var. And my next project will go with ENV var from the start.

> 12Factor Nbr 11 : "each running process writes its log, unbuffered, to stdout"

Yep, give me 5 minutes to do that.

When I read 12 factor, and decided to give it a shot, by the end of the week, my app (2 devs full time for 2 years) got nearly everything nailed down and I was able to benefit from it. The last remaining points were harder to implement but I undestood what I needed to do and why.

Most of the advices of the 12factor manifesto can be applied to any programming language/framework/architecture/buisiness and make the deployment/management of the app at least easier.

When I read your manifest, I did not understand most of it. Even your examples do not "talk" to me :

> In your food ordering workflow, instead of writing a payment processing microservice that captures different failure modes, write a payment processing function that processes a payment or fails. The event system should capture the retry or failure event which is processed thereafter. This ensures your business logic is simple and easy to scale.

What event system ? how does it capture the failure mode ? I have an app that is now 4 years old, on which 6 devs are working full time, what can I do today to make it better in a month or two ?

Why are your trying to force me to use GraphQL and make a tech choice for me ? At no point 12factor says use "HTTP REST JSON API" even is a lot of user will go that road, it's their choice and not forced by the maifesto itself.

https://3factor.app/#comparison-to-12factornet

It's an architecture. No claims were made that switching architectures is easy, quick, or even recommended.

There should be no comparison to 12 factor. The choice of title did make me click on it though.

As mentioned in the article, it's an architecture pattern similar to Redux but applied to the whole system, ie Action -> State -> Event -> Reaction

It's a shame Redux uses the word Action instead of Event because I think we'd all have easier time discussing Event Sourcing right about now

It's because Redux was specifically written as "just another library implementing the Flux Architecture", and Flux used the term "action". See Dan's comments at https://github.com/reduxjs/redux/issues/891#issuecomment-147... and https://github.com/reduxjs/redux/issues/377#issuecomment-126... for justification and links to prior discussions.

> What event system ? how does it capture the failure mode ? I have an app that is now 4 years old, on which 6 devs are working full time, what can I do today to make it better in a month or two ?

Good question. Definitely the wording can be improved. Let me try to explain though.

Suppose you write a simple function which processes payment. It fails. Now you can insert a state saying failed with error code 1001. Now this insert creates an event which invokes a serverless function specifically aimed at resolving this error.

This way you have smaller cohesive functions which do "atomic" work as much as possible.

PS: Co-author here.

What is a "serverless function"?

Something along the lines of AWS' Lambdas

I’m guessing they mean something like AWS Lambda.

Perhaps a guide on moving from 12-factor to 3-factor apps would be more clarifying.

So let's say an event is emitted, does this happen?

* "payment processing" handles it -> changes state -> another event emitted -> "restaurant approval" processes it -> changes state -> another event emitted -> "delivery assignment" processes it

Or does this happen?

* event emitted -> "payment processing" and "restaurant approval" handles event asynchronously, -> state changes -> event emitted -> "delivery assignment" handles event.

In this particular case, I meant the former. But you might have workflows where the same event is delivered to multiple serverless compute things simultaneously.

Oh. I see now. Thank you for posting this.

So instead of clear code to handle the error you have a dozen logic components watching for some events and modyfing state again triggering events and modifying state. Yeah, i absolutely can see how this approach will NOT increase maintainability because now you have 84 logic functions watching for some graphql events, and they may never be down for a microsecond because they would miss a (critical) state change: graphql subscriptions are not like persistent queues

That is why events must be persisted and reliably delivered. And in the frontend, you must subscribe to the state of the world and not necessarily each state change. This way if frontend goes down and comes back up you can show the new state instantly.

Exactly. We created our own even store on top of a SQL database. It's quite simple to do, and it is blazing fast because it is custom built for our use case. It's an append log of every event, their time, their type, and some other info about them (account, user, etc.). The event itself is stored as json, and we can just use Json functions of SQL Server to query those for diagnosis if need be. We've also got failure reporting built in, and strict referential integrity for various other tables to link into the main event storage table.

It was actually a pretty fun weekend project.

Right. But, what if the client (eg browser) would ask the system if the message has been fully processed? Then the client can re-send the message if the Server lost the original message? For example:

  client: "Hey, buddy! Did you process that message I sent you? Here is the ID."
  Server: "No. Send it again."
  client: "Ok. Here is the message again."
  [Server processes it, completes processing it, sends back response.]

If done this way, wouldn't you be able to reduce the need for persistence of the original message and events from the client? (Not that persistence of the original event on the Server would be a bad thing.)

We can do this via the store's aforementioned failure reporting. I guess it could be better? I prefer to have a log of all messages since forever, as in Event Sourcing.

Seems like that is more trouble than it's worth to try to make everything happen purely on a message bus, though the mechanism for letting the system know a new event arrived is a message bus, it just includes the message id, for us.

I’m waiting for 1factor apps.

"Just write teh coad"

I've never used GraphQL, would using it mean I do not have to define DTOs or other API objects, I would just create models from whats been passed into GraphQL?

Talking in a Java context here.

What would you use nowadays to actually implement the real time GraphQL subscriptions and event system in practice?

We built https://github.com/hasura/graphql-engine , https://hasura.io that sets up the plumbing for this architecture with state being Postgres.

There’s a lot of other tooling emerging in this direction too. For example, set up a GraphQL server and subscriptions with Postgres and listen/notify and use SQS with lambda. Or dbezium + Kafka + custom service for event delivery.

“3factor” strikes me as a bad name. In IT, 2-factor universally refers to login security/multi-factor login. “3factor” sounds like you’re adding another security method to the login screen.

It's a play on "The Twelve-Factor App" https://12factor.net.

https://3factor.app/#comparison-to-12factornet

This may be silly, but I think copying the format and spelling out the number might make this look less like "2-factor":

> The Three-Factor App

"3 factor app" immediately clicked with the 12 factor app with me.

I was hoping on a reduced approach of the 12 factor app due to modern developments.

But they mention they are different factors from what 12 factor describes:

> The 3factor name is inspired from 12factor.net ... Although the name is similar, 3factor.app is actually an architectural pattern.

So you still need to pull in some of the 12 factors after you're done architecting your code to 3 factors.

Reply on news.ycombinator.com