MiniCouchDB in Rust
garrensmith.com
garrensmith.com
In general, we are trying to move away from running your whole application in CouchDB. We would prefer that you have an application layer in front of your CouchDB instance. We recommend that you put up a proxy if you want to expose your replication to PouchDB. That way you can add security around that endpoint.
I can't see someone making a meaningful document level ACL outside of the db without some serious effort, nor would I consider home-baked external authentication for replication simple for the same crowd.
A side note I don't think it's good practice to expose your db to the internet, but if you look at Mongo's Atlas this doesn't seem to be as big of an issue as it may have been in the past.
Syncing was a solved problem in the CouchDB world, so we moved to CouchDB.
Access controls also seem like they should be a “solved problem,” something we can adopt that works well with CouchDB.
Is there some plan for a standard ACL approach for CouchDB users? If we need to put it into some app in front of the database, that eliminates the “solved” nature of sync... or at least introduces added complexities like a reverse proxy.
I really hope CouchDB could bring in some ACL management into the ecosystem as a standard, drop-in sort of approach, and not something that requires custom development for every single app that wants to use CouchDB / Cloudant.
I think it's generally a problem that taken alone these features aren't useful until they are all together good enough to invert the system such that the DB answers and occasionally proxies instead of the other way around.
In all fairness, they did have an auth story, right, and recent documentation suggests they reconsidered that path and now suggest keeping it out of couchdb. So, to me, this says auth was something they never could get right because it is complicated. And I took that a step further to think that it's better to have it outside because you can use any auth solution you want, instead of what the couchdb people felt were the best way to do it (it's smart that they realized they got it wrong and changed course).
I'm confused why everyone here seems to think reverse proxies and auth proxies are complicated. Isn't it the case that all apps of any complexity are a bunch of small services wired together behind a proxy? My auth proxy is all of 50 lines of code, my reverse proxy is 8 lines in nginx conf and it's all held together with a docker compose file that is declarative and works locally as well as on my production server.
The problem with auth is in the authorization realm, not the authentication realm. It's super easy to do authentication in nginx for example using client certificates, basic auth, api keys, or in your app layer via checking a separate DB or cache for tokens, say via a JWT.
However, it's not trivial to say user "a" owns document "b" but not document "c", and that user "a", in fact, shouldn't even know that document "c" exists at all, while also maintaining the replication that CouchDB has built in.
What we want is one single DB that has all documents, and can replicate all documents on the server level with another DB, but can expose a user-specific changes feed that replicates only data that user has access to. I should be able to grant / revoke access to a document at any time and have it propagate, and all documents should be owned only by the creating user by default.
The choice they are requiring us to make is to shard data ourselves by user (which is unusable in the context of 2 users sharing data), or implement a separate layer that can understand CouchDB replication and can do the filtering of the changes feed as well as the write access to ensure that the documents are restricted correctly.
Take this a step further and now that I have to implement my own authentication AND authorization outside of CouchDB and protect it from the end user directly accessing it, and you could ask the question of why do I even need CouchDB then? Why not just speak CouchDB replication protocol on top of my own auth database and store documents there too?
All those things you note are important, and I just can't see how CouchDB would get those things right inside CouchDB.
My impressions with Firebase/Firestore were that they tried to do that with their "rules" system, and it was a not-quite-JS declarative system that relied on you understanding the non-standard parts of their auth system and the things they exposed. I always felt like this was going to be a huge hole in my app and I would have no way to validate all the edge cases. I feel like this is a complicated beast to do in a generic way and CouchDB was smart to leave it to the app developer, rather than the devops/sysadmin role.
Aren't you being a little stingy with your appreciation for the sync part of CouchDB/PouchDB and a little bit overblown with your worries about understanding the CouchDB "protocol?"
Replication and sync are really challenging problems even if you just think about sharing data in two places, and when you start having your JS code deal with revisions, it gets messy really quickly. That's what appealed to me about PouchDB was never having to really think about sync, other than how to handle conflicts.
But, the CouchDB protocol is just HTTP. And, making a proxy to talk to that is as simple as importing a http proxy module. You are just responsible for your authz logic, which is hard, but at least you can make it exactly the way you want it. I don't really see how mapping your authn logic onto standard HTTP verbs like POST, GET, PUT is that complicated.
Having said all this, you clearly have thought through this stuff deeply, so I'm very interested in hearing your thoughts here because I'm sure my comments are wrong past the surface.
I feel that moving further in this direction to make couch a competitor at the DB later will be the death of couchdb in the long run, because it is removing the only real advantage it has over its peers, which is https based replication to the browser and between peer DBs, and the fact that it’s built on Erlang and can handle replication to many peers with ease. If you make us proxy and wrap the Erlang app in our crappy (excuse the language) business layers like Java and C# to build the proxy ACLs against a separate DB, you’ve totally covered up the huge benefit of the safety and concurrency of Erlang in handling the replication to endpoints for us.
I suppose it could be related to their use of mochiweb still for the web layer. Maybe they'll add on HTTP/2 eventually and no longer recommend a reverse proxy.
There are alternatives for HTTP/1.1 and HTTP/2 in Erlang, like Elli (http/1) https://github.com/elli-lib/elli and Chatterbox (http/2) library https://github.com/joedevivo/chatterbox
https://github.com/daleharvey/noted/blob/master/index.js is a very simple example of how it can work
I acknowledge that this is the same issue in a traditional dbms, but it doesn’t try to handwave away this complexity from you. It doesn’t show examples on their site about how easy replication is to set up only to be betrayed later After you’ve already integrated and marveled at the couchdb sync performance when you can’t build real permissions and would’ve been better off using Postgres and getting a bigger community and ACID/CP. and let’s be honest, almost every app is gonna need a rdbms at least for transactional data anyway (we know you’re not storing stripe billing records in your couchdb) so then the question becomes, is the sync protocol even good enough anymore to warrant using couchdb?
Case in point, this example code has a massive security issue that allows anyone to impersonate without tokens if there is an active authentication request open. Hopefully no one has used this example code to build a production system that has real user data.
This is a perfect example of why this should really be part of CouchDB/PouchDB itself and not something each person must write themselves. This should be solved once, solved right, vetted by the community, and be easy to fall into a pit of success.
I really like CouchDB and PouchDB as a product, but this insistence that this is the right path is really holding you guys back.
Both of those features are purely a client-side concern and exist because the original intent of CouchDB was real-time replication to browsers. Otherwise the proxy could do the CORS as well as the authentication and CouchDB would only require an http-level authentication pattern (like basic auth).
Having said that, I do agree that this should be compartmentalized and the end user should be able to pick and choose what features they want to allow, but I don't think that this should continue to be a separate concern that everyone is building themselves, it should be a first-party solution.
Right at the top it says “Make the db-per-user pattern obsolete.”
Db-per-user enables trivial syncing between multiple devices for a single user. That means someone’s tablet, phone, and laptop can all stay in sync thanks to CouchDB’s replication. And we can do it without building anything extra, as long as we have a client library that talks CouchDB’s http API.
Does this new approach eliminate that possibility? Will we now need a server in front of the database for that?
That’s the main issue with this whole thing. The solid replication and change events and conflict resolution all together is the only way that couch is compelling at all compared to other dbs in my opinion, and having to partition the db to the user level for security without adding a proxy just defeats the whole purpose for most use cases.
One of the things that really turned me off from firebase is the rules system where you have to build an ACL there. It's been a few years since I've played heavily with it, but I really disliked trying to essentially build my authorization system into a backend which I couldn't run locally and was primarily edited in their web console.
For me, writing a proxy that sits in front of couchdb is very simple. I use JWT tokens that get passed between components in the system. I keep my authorization logic out of couch. I can write normal unit tests on my proxy. And my pouchdb client code mirrors the structure of my backend structure, which is a mental model I really prefer.
I guess I'm saying I think having a separate database for each user makes more sense to me, not less.
The only thing I'm struggling with is running code when a document is updated. I have a polling client that watches for the _global_changes updates, but it seems really hacky. I wish there was a better way to get access to all database changes that looked like firestore functions.
Still pretty cool, do you intend to develop this further?
I expected some kind of live demo, so I enabled Javascript. It turned out to be a static page with text. Not even a toggle-able menu on narrow screens, just a static page.
Why does it need Javascript to render a static page?
If you were inspired by mini-redis, why not go the extra mile and turn it into another great educational resource?
I wish I knew rust and could say something more substantial lol.