AuthZ: Scalable permissions system
medium.com
medium.com
It seems that the scalability limitation is the same as that of every graph - the efficiency of scaling a graph beyond a single node.
Not bashing what you are saying, just want to understand what makes it "highly scalable", as it isn't clear from the article.
Also, I was wondering: Are ALL permissions stored as an Object::relation pair? i.e. do you need to register permissions for all new entities relationships, or do you have some way of storing more dynamic permissions?
The idea is to put all data in one place, and then aggressively optimise for answering queries about that data. For authorization you're pretty much always asking specific reachability questions. That's where the design of the index comes into it.
> Also, I was wondering: Are ALL permissions stored as an Object::relation pair? i.e. do you need to register permissions for all new entities relationships, or do you have some way of storing more dynamic permissions?
In Zanzibar, there is a configuration format for computing relationships dynamically. E.g. "anyone who is an editor of a document is a viewer of a document". I'm not sure if they implemented something like that.
If you're interested, I wrote a guide on relationship-based access control that includes a section on how Zanzibar(-like) systems fit in: https://www.osohq.com/academy/authorization-academy-chapter-...
> just want to understand what makes it "highly scalable", as it isn't clear from the article.
While this post might not cover in depth why their system is scalable, the Zanzibar paper[1] and this post[2] about Zanzibar covers why similar systems are scalable. A oversimplification would be to say that finding a path between two nodes in a graph can be recursively broken down and performed in parallel -- the Authzed implementation looks roughly similar to a map reduce.
>Are ALL permissions stored as an Object::relation pair? i.e. do you need to register permissions for all new entities relationships, or do you have some way of storing more dynamic permissions?
Zanzibar implementations do have graph schemas (called Namespace Configs in the paper) which offer set operations for computing dynamic relationships. When you create a new user or object in your app, you typically also write a few relationships into Zanzibar and the rest of their permissions are implicitly granted through the computed definitions in your schema.
If you're interested in learning more about Zanzibar, we're giving a PapersWeLove talk that will be streamed on Twitch next week: https://www.meetup.com/papers-we-love/events/278148236/
[0]: https://authzed.com
The discussion on Flattening is super important, though for most teams, I'd also add this dances around the costs of an extra piece of intertwined DB infra.
For our internals, we ended up having to reject OPA and friends, including what Zanzibar does, because we decided our customers wouldn't want us to spend our infra innovation tokens on an extra DB and figuring out how it should intertwine on the hotpath for queries to the main DBs. For bigger/heavier sites like Carta (with especially complex rules) and especially high-traffic ones like Google, as hinted by the percentile charts, I can see their decision making way more sense. For most consumer & b2b sites though, I'm not as convinced, and especially long-term.
We ended up going with Casbin as it can do ARBAC policies + reuses our SQL DB. So it's more like adding a lib than infra, even though it does come with policy features like an analysis-friendly DSL. Likewise, that means we get a few ways to tune bulk queries: most are automatically fast by smart default roles & logic, some via special-cased app-level opts (so out-of-DB), and if we ever need to, we can always do optimized in-DB queries.
The options for policy engines for b2b apps is surprisingly slim. The whole experience has made me think that a SQL DB with a strong multitenant+arbac RLS story for Django/Rails/etc. users and a cloud autoscale-to-zero capability will make a killing for b2b SaaS's. Amazon Aurora and friends are close, but feels like it's still another couple years in practice.
Was this problem solved? Is AuthZ open source? (I couldn't see a link to a repo anywhere - but am reading on mobile and maybe I missed it)
From the screenshot [1], it looks like it's hosted within some larger dashboard system – does anyone know if that's OSS or something custom?
[1]: https://miro.medium.com/max/700/0*JATQRwLPT_MgrMHT
EDIT: Ah, looks like Backstage – https://backstage.io/demos
I noticed they’re loading third party JS with at least one postMessage handler with a wildcard origin. They were also sending the current page title (which can include things like company or investor name) to this provider. It appeared they were also using a version of Knockout.js that is 2+ years old. [1]
This did not inspire a lot of confidence. The linked blog post at least takes security seriously, but it’s clear they’re patching over a lot of legacy cruft.
There’s a hell of a lot of sensitive data on Carta, which I’m sure they’re already selling to their “partners” anyway, but IMO it’s only a matter of time before they (or one of their “partners”) are hacked and a whole bunch of companies suffer from information breach.
IMO the main reason they haven’t been hacked yet is simply because you need to pay $2k just to get access to their questionably secure app.
[0] Your lawyer tells you that you need a 409a and Carta is the best. But if you only want the 409A, and don’t want to get growth hacked onto Carta’s platform, you’re out of luck. They won’t do the 409A for you unless you complete the full onboarding process, which includes appointing Carta as your company’s transfer agent, and uploading the email addresses of your investors and allowing Carta to spam them with tasks like re-signing legal documents that you issued off-platform.
[1] This is not a technical analysis and I’m just going by what I remember from watching my local chrome dev tools. Don’t @ me.
Considering the fact that 409a valuations pre carta would cost $10k, $2k doesnt sound like a such an unfair deal. You could always go the legacy lawyer companies route and pay way more for a single 409a valuation.
As for onboarding process - you do need a hell of a lot information to conclude said valuation.
Take a look, I'd love feedback and to build it into something valuable for the open-source community.
I've made this mistake more than once when designing an authorization system. Sure, you can build a fast mechanism for answering the question "does actor A have access to object B" - but can you efficiently answer that for 10,000 objects at once?
Say for example you're building a search engine, and you want to search across all public data but ALSO all of the data your user has access to that's private. You probably can't run the search, get back 10,000 results, then apply your "can X access Y" logic to all 10,000 of them.
(I'd love to know how GitHub code search implements this - it shows me results from my private repositories mixed in with results from their public index).
Full disclosure: I'm a founder of Authzed (YC W21), which is a productized Zanzibar system.
We have three main concepts in the sytem:
- Capabilities (eg. "read profile")
- Selectors (eg. "profiles belonging to customer X)
- Groups (eg. "profile")
All queries are of the form "Do I have capability X on resource Y" (where resource Y belongs to a particular group). A "permission set" is a mapping from a capability to a set of selectors.
Selectors can be composed (eg. you could OR two selectors together) but are predefined. Since they are predefined, our search implementation can understand a subset of the selectors, and translate them into an ElasticSearch-compatible query.
We then filter the results by evaluating the permissions normally: this catches any cases where there was no way to construct a query, or if the constructed query could not be completely precise. If this occurs, then we show a message on the page like "some profiles were hidden because you do not have permission to view them" and then we might show 19 profiles on a page instead of the full 20, so that our pagination can still work efficiently.
Aserto looks very promising (https://www.aserto.com/), but it still in private beta.
It's crazy this still is part of the stack where there are no great solutions.
Seems like a few others have come to the same conclusion :)
We're working on this at Oso (https://osohq.com) - I'm the CTO. Oso is an open source framework for authorization. Policies can reference application directly, so any authorization decisions can be made dynamically based on the data. And your data doesn't need to leave your application.
> Often the issue is the inability to easily push down filters into the database or search index.
We actually support pushing down filters into the database directly through a feature we call... data filtering. We support Django and SQLAlchemy today (https://docs.osohq.com/python/guides/data_access.html) but will be making this available for non-ORM/non-Python users in the next few months.
It's also based on Zanzibar, and it's used in production applications today. Cloud version coming soon (using cockroachdb) or use the open source version (postgres).
agreed that it's wild that this isn't solved for b2b apps in 2021. the 80's were when most fundamental innovation happened here, then fundamentally more flexibility in the xacml days (~arbac), yet here we are...
It's indeed a hard problem, because you want the solution to have the operational characteristics of a library (millisecond latency / 100% available to the calling application), but at the same time, give you the convenience of a service / control-plane (allow you to centrally manage the lifecycle of authorization policies, sync properties about the users to the edge, drain all the decision logs back to the control plane, etc).
It's one of the most frustrating wheels that developers still have to re-invent, and it's great to see more companies trying to innovate in this space.
As @tristanz mentioned we're in private beta, and happy to engage with folks that want to solve AuthZ :)
Also, the full Ory suite is very well developed and worth looking into. Fully open source!
The most frequent design mistake I see is an engineer with a "hammers before nails" complex. They want a perfect design that will work for everything (that they have so far thought of) in some beautiful and hidden way. So they build the thing, and they think it's great. And then they go to hook it up to products... and it doesn't work well. The user story, the context diagram, and the mock-up all need to come first, not the code.
OPA is one solution that's emerged in recent years. The author describes his team's point of view after evaluating it for their purposes.
Oso (https://www.osohq.com/) is a framework for application authorization. Disclosure: I'm a cofounder. Based on the post, the Carta team started their work in 2019, but we only open sourced Oso mid-2020, so they wouldn't have known about it.
And in the last few months, a few other companies have announced offerings to address this problem too.
So it's understandable why Carta chose to build. Now/soon other teams like them should have more options to choose from when they encounter similar challenges too.
You'll often hear people say "you should never, ever build something if you can buy a solution instead" (where buy also includes using an open source package) - but there are actually a bunch of other things to consider:
- Is the solution you're considering well maintained and likely to keep working? Open source projects sometimes stop being updated. Vendors go out of business.
- What's your escape strategy if the vendor DOES go out of business, or massively increases their pricing?
- Integrating with a vendor / package is often a non-trivial amount of development work. Have you considered this relative to building your own, custom solution?
- Do you have specific requirements that go beyond what's available off-the-shelf?
It's absolutely true that your engineering efforts should be spent on the things that make your organization unique: time spent on your customer's problems is more valuable than time spent reinventing a wheel.
But sometimes, the available wheels really aren't fit for your purposes.
I've built when I should have bought, and I've bought when I should have built. Getting it right is hard! Often it doesn't become clear if you made the right decision until several years after you've made it.
Also you avoid vendor lock-in. Even Open source projects cause lock-in especially if they are overcomplex and fast moving.
Carta seems to have an extremely complex authorization scheme but I wouldn't say that's the common case. Many organizations can get by using simple role + account based permissions.
If you are a member of this account then you have access to the resources this account owns minus anything you can't access because you don't have role X or Y. Oftentimes the account based permissions boil down to where clauses in SQL statements. The role based permissions can be mapped to scopes in a JWT or similar.
Anything beyond that can be done with if statements in your "parse user input" function or simply caught later on resulting in a 4XX error.
I know at my particular organization, even when we have cross-account sharing, it's heuristic based. It's simple enough to just apply it in the application code. Stuff like "did this user's organization receive this shipment?"
I feel like you have to be either a very large organization trying to make sure people do things consistently or have a very, very granular permissions model to even embark on making the decision of "build a whole authorization product in house or outsource it?"
> During our discovery process, we evaluated both open source and vendor products.
> But we had a problem: There weren’t any open source implementations available. We decided to build our own system, adding several of our own modifications to make it an even better fit for Carta.
There's no mention or implication that they'll release it as open source, this seems to be an engineering blog post (for hiring lead gen or building the engineering brand of the company or whatever).
- A: But how do you solve authz?
- B: authz
- A: Yeah, how do you do it? What system are you using?
- B: authz
- A: ... Why are you not answering me?- Shiro covers authorization, plus authentication, session mgt, etc. The Carta project is authorization only
- Shiro relies on external data to make authorization decisions. The Carta project stores that data internally
- Shiro is run as part of your app. The Carta project is run as a separate service
- Related: Shiro is specific to Java.