Google Cloud Firestore NoSQL database is in GA | Hacker News Reader

Google Cloud Firestore NoSQL database is in GA

cloud.google.com

283 pointsmarkoa154 comments

There is something i don‘t understand about Firestore: If you use the web client your javascript code is directly communicating with the firestore database, you have no control about some user specific limits. How do you prevent some rogue actor from effectively doing billions of reads or writes. With server side logic you can implement rate limits but not with a fire store web client. The only thing you can do is to limit your monthly budget? So one bad guy can burn all your money for a month without you having any possibility of limiting it?

I assume that there would be some rate limiting and anti-abuse measures here, like most Google products have. It’s a good question though.

How would an automatic system even prevent unwanted deletion of database entries or that users access others' data? Maybe I am lacking imagination but without server-side validation the practical use cases seem very limited.

It has this: https://firebase.google.com/docs/firestore/security/rules-st...

Ah, interesting, so you basically put the server side validation code into the database. And nice, it can even handle authentication.

Cloud firestore supports placing rules on the database to prevent access by users that aren't authorized. You can independently control get, write, update, list, etc. The way the rules work can sometimes be irritating, in particular for list queries, but they do work.

In terms of rate limiting the only one I'm aware of is writes to a single record are rate limited. This impacts normal development when making counters that need to be updated, which leads to an odd "sharded counter" model.

When I last looked at Firebase, it seems like they bake auth into database so in this case a user is limited on what data they can read, like a select where uuid === uuid https://firebase.google.com/docs/database/security/user-secu.... And I assume non-authenticated read requests don't have access to everything...

Think of it as an API layer and database baked into one.

This is a good question. I think a more general feature it's lacking is auditing (as far as I've been able to find), from which you could judge whether a user was abusive and revoke permissions. There's profiling tools, which you may be able to hack together something for auditing if you were so inclined: https://firebase.google.com/docs/database/usage/profile

In these kind of systems, your app users also become database users and effectively each visitor gets their own token to access the database.

You then add rules to limit users so they can only read/write their own data. This works fine for many simpler scenarios and avoids the need for a backend completely, with the potential risk for overuse.

If you need more control and protection then you should have a backend layer to get the data server-side.

By "In these kind of systems, your app users also become database users", do you mean by RBAC or some other session-based control on the backend, or does Firestore perform some intelligent user entity resolution I'm missing?

As far as I can tell, any naked frontend-only client serving content without its own access management sounds quite open to a single bad actor spawning a large number of sessions on the same host, and on a small number of distributed hosts, to eat up all available resources. I'm not inclined to ever trust a number of users behaving nicely at n>100 personally.

Given that Firestore appears to advocate "serverless" client-driven applications, unless there's some foolproof DDoS mitigations I'm finding it a hard sell, especially on the whole "if you need more control and protection, do X" argument - you rarely need that until somebody straight-up exploits you, and when you do, you're not going to be particularly sympathetic towards Google's marketing speak.

You can require login for read/write access

I struggled with this. We implemented Firestore (successfully) in a production mobile game for the real-time chat and configuration aspects. The SDK WebSocket implementation for real-time updates was a huge factor here.

At the time we were using the Firestore beta, Security Rules were also very poorly documented with no tooling apart from the editor (beta product - I know, I know).

Ultimately we found several vulnerabilities in our own implementation of Firestore that were difficult or practically impossible to resolve on our end. The workaround was to communicate with Firestore only by using the admin SDK.

I still completely vouch for Firestore and Firebase as a product.

You have control over user specific limits. You can use Firestore Security rules, and if that doesn't do it you can create a service that runs server-side code to do what you want and then have the client hit that service.

> You can use Firestore Security rules

I think the main problem is that they bolted security (access control) on top of their existing system, rather than make it an integral part of the design.

We built it specifically for Cloud Firestore and as an integral part of the system from the beginning. We always like to hear where we can do better to help prioritize the next set of improvements, and would love to know what gives you the impression of being bolted on.

I’m building an app at the moment with Firestore and 99% of it is amazing or better. It’s a really fantastic tool and y’all are enabling people like me who are less adept to make good stuff. So thanks!

Two minor notes (one on security rules):

(1) in the tutorials I’ve seen, the security rules have received little attention and the documentation I’ve read is a bit confusing about the spectrum of possibilities for security rules. Tbh, I’ve punted security rules altogether until I’m closer to “beta ready” as opposed to writing them along the way. I suspect this is bad practice but haven’t found a better way. Much of this is my fault lol.

(2) “presence” systems still require the use of Real-Time DB. +1 feature request to support user connect/disconnect events in Firestore.

firebaser here Completely agreed that we should do a better job in documenting the security rules. They're very powerful, but a huge paradigm shift for most developers, one that is too easily ignored (at your own and your user's peril). We've already released a bunch of improvements on the documentation, and continue to work on adding new documentation, video tutorials, and console improvements. Please keep telling us where we can/should improve security rules further though.

And yes, also agreed on needing a solution for presence system, either in Firestore itself, or separate from it.

That would be a really weird user experience given how long this has been in Beta. Google has had over 1.5 years to get access control right, so I bet (hope) this is not an issue for new users.

Limiting writes would be fairly easy. You could have an onWrite Function trigger[0] that writes activity to a rate limit Firestore document. When new write requests come in, the Firestore rules can read that rate limit docuemnt and decide if the particular write should be allowed or not.

Limiting reads is interesting. I think generally the idea is data is cheap enough that most people don't have to worry about it. If you do you could gate access through an API in a Function.

[0] https://firebase.google.com/docs/functions/firestore-events

[1] https://firebase.google.com/docs/firestore/security/rules-co...

For a subscription based Chrome extension (https://www.checkbot.io/), I use Firestore, Firebase and Cloud Functions to handle subscription activation and login/authentication. I use Paddle for payments which fires a web hook to a Cloud Function when a purchase is made.

Works great so far for no cost and it hasn't required a single bug fix or any maintenance since launch. Firestore definitely has a lot of surprising caveats though and you need to design your app and data around this to avoid trouble later.

Does anyone have any stories to share about when they outgrew Firestore, what they migrated to and how? I wouldn't be keen to use NoSQL if my data became more complex.

Interesting model, do you have an OSS example of something like that?

I couldn't find one for Paddle to use but I'm sure I saw a few for Stripe. I wanted to use Paddle because they do EU VAT for you. I could be tempted to open source it though.

The gist of it is when a subscription is purchased a web hook triggers a Firebase Cloud Function that stores the subscription data (i.e. email address + subscription expiry date) in Firestore then when someone logs in via Firebase authentication you check if the email address is linked to a valid subscription. When the next payment is made to the subscription, you bump up the subscription expiry date. I've found there's little that can go wrong once it's up and running.

I'd personally be super interested in that code, even if it wasn't usable out of the box (i.e. just dump what you have)

Would also love to hear more about how this is done... I'm in the process of setting up Stripe subscriptions on a website, but have been scratching my head over the best way to accomplish everything without setting up my own back-end server.

I'm an engineer at Stripe and thinking about this problem - drop me an email? In profile. Would love to run a few ideas past you (or others frustrated by this).

That's really useful, thanks. As is the pointer to paddle, I think they may have been missing some features last time I checked them (some time ago, or I just missed it) but that was a long time ago and they seem to have everything covered now. Fantastic! Saves some code I've been putting off writing.

Paddle's pricing seems a bit high at 5% + $0.5. Is there a reason why you opted to use Paddle over Stripe? Curious because I'm similarly working on a Firebase MVP at the moment and looking to integrate a donation feature.

EDIT: nvm, you answered the Stripe vs Paddle question in another reply.

It does seem high but even doing tax returns is annoying enough so I've no desire to get involved with EU VAT and tax issues selling to other countries. This way I can set up payment and it runs itself which is worth the cost and Firebase means I don't have to think about DevOps too. Gumroad is another payment option to consider.

Interesting:

> Stripe does not calculate any tax information for you.

Paddle:

> We automatically handle the calculation, collection and remittance of taxes in every single country on your behalf: you don’t need to do anything.

Paying Firestore user here.

1. Disappointing that it has gone to GA without providing proper search (after a long beta). Can anybody explain why a service run by the worlds #1 search company continues to point users to third party services if you want to implement a basic text search in a database? I genuinely don't understand that.

2. Feature requests (or complaints) re backup, queries and documentation are not new. Nor are the answers (or excuses) we see here, which revolve around scalability and (until now) being in beta. BUT all users I have ever heard say they use the product for speed & ease of setup & convenience - for MVPs, not for massive googlesque data. It almost feels like the product market fit is not quite right. So, forgive my technical ignorance here, but worst case scenario, why not provide the features with the caveat that they are slow or expensive or won't work above a certain size of db? Isn't half a solution better than no solution?

The database and whole Google/Firebase app suite thing has some strong points. But to be frank, and I'm sorry if I'm being dismissive of hard work and technical wonders here, from the perspective of a customer and outside observer, a number of things smell quite off.

Yep, also disappointed to see it in GA missing queries and back ups. Makes it a non-starter for most folks.

Curious - what do you use? any suggestions of good serverless NoSQL? with the built in auth and extras Google/Firebase offer? What about relational? I've used Back4App, but with it being small and FB having killed Parse, it wasn't right.

Have you looked at Sanity [1]? You get both a fast cloud backend (with a powerful GraphQL-like query language with joins, transactions, object-level security, text matching, real-time change notification, etc.) and a collaborative editing front end (optional, but recommended!). The latter is open source and modular. You can deploy serverless just like Firebase/Firestore. Gatsby is a popular way to serve Sanity.

Disclosure: I've contributed to their database tech, but I don't work for Sanity. Mostly just a fan.

[1] https://sanity.io/

Check here: https://cloudbackend.appdrag.com

I'm looking at FaunaDb very closely, seems like a nice "all you might ever need", but a bit unfamiliar at first glance.

They keep promoting Algolia. Actually once of the firebase engineer mentioned why to build search when already Algolia solves it. I was like, wow.

Thank you for being a paying user.

1. Mainly because it's both a different search problem (general DB vs specific to web search) and hard engineering wise given our model; we implement not only the cloud database, but embedded versions for iOS, Android, and Web - not to mention real-time functionality and tailoring it to how our index engine works, etc. While we have a lot of customers and use-cases that don't need Fulltext Search, we totally agree it's important and have done explorations on how we'd deliver something along these lines.

2. Agreed. During the beta program we've delivered the managed export and import service for backups, adding array contains capabilities to queries and have got close enough to delivering Collection Group queries to mention them as part of GA. For documentation our tech writing team as done a lot of updates, new pages, and fixes - we know there is always more to do. Cloud Firestore is definitely used in production and at scale by our customers, and with nearly 1 million databases being created the range of use cases and traffic/load patterns has been vast. Our beta program involved working with a lot of them to improve things like hardening and scalability to ensure we can meet our 5 nines of availability SLA.

"Isn't half a solution better than no solution?" -> In a lot of cases, absolutely not. A half solution that falls over when you tip a certain point of scale can result in extended downtimes, since the solution often ends up being "we need to completely rearchitect this", which isn't easy or quick when your business is out of commission.

"from the perspective of a customer and outside observer, a number of things smell quite off." -> Sorry to hear this, I can only hope the continued hard work from the team will turn you around.

Thank you for the answer. I have to admit it doesn't quite sway me, for reasons such as those below, but thank you.

e.g. yes I realise they are different search problems, but I'd presume that Google is nonetheless well-equipped to handle the document db one. the only apps I can imagine that couldn't benefit from a search box are games - anything content-focused or ecommerce focused needs one, and the majority of utilities benefit too (yes I can do chat without search, but it certainly benefits from being able to search through chats) - any examples? yes I realize having to do Backend/iOS/Android/Web is hard (as it is for everybody else), but with on device cases at least the db is smaller. Im sure you do have big users, I didn't mean to imply otherwise, but with my admittedly very limited knowledge I'd still wager that a majority do not see uptime and scalability as the most urgent improvements, but rather those we are discussing. In our case, give us just 2 9s of uptime, and give us the above queries and searches even if 2x as slow and expensive as you'd like them to be, and limited to a db the size of an average relational db, and that would beat the extra 3 9s of uptime and the super scalability any day. Not least because, I don't mean to be rude here, just candid, but if we were ever to reach a point where we needed that massive scale and uptime, I'm not sure I'd be keen to trust Google with user data.

To be clear - I like several things about Firebase/Firestore, which is why we use it and why I'm insisting on badgering you here. I just wish I could be completely comfortable with my choice rather than wondering every day if I shouldn't just use something else.

Been loving Firestore! It has been my first real experience w/ NoSQL in an MVP to production-ready quickly. It's been SO easy to experiment with and learn. Community has been great.

Gad you been using realtime database before? Curious about experiences with transitioning (additional fetaures do sound appealing).

We were using RTDB and migrated firestore. The querying limitations in RTDB is really painful. Transitioning was not that difficult. Basically wrote a script in node.js to migrate the data. (our db size is <100M)

I'm guessing this includes updating your source code for the new library/methods

Absolutely. Need to go through a whole bunch of changes (in architecture, query, security rules, backend like cloud function etc...)

Anyone with inside knowledge of Firestore care to comment on the ETA of more advanced query functionality?

We have played with Firestore quite a bit, but rely heavily on the ability to do aggregate queries. Reading all of the documents and performing this on the client side is nowhere near good enough. Nor is triggering functions to update a "count" or "sum" property on a doc.

Edit: Looks like a PM answered on another thread...

"It's a point of internal discussion on scalable ways to achieve this, but nothing we can promise. We definitely see the need for it."

The documentation clearly mention those features aren’t supported as of now.

The Firebase team is aware of that need but it’s very hard to deliver such features at scale.

Firebase is used everywhere these days , so a single change in the platform has impacts in petabytes and massive thousands of cpus load.

I get that it isn't supported. We have followed the recommendations mentioned in the docs but found the usage of these workarounds to be sorely lacking in performance and reliability.

Firestore is very limited here. I trigger a cloud function on every new entry and replicate my data in bigquery. This solves the problem for me.

Firestore was a big disappointment for me.

By far the biggest pain point of the RTDB was its poor querying capabilities. Firestore solved some of those problems, but it's still very limited.

We moved away from Firebase primarily because of the serious limitations of the 2 DBs.

Product manager for Cloud Firestore here. We extremely grateful for all the amazing beta program customers and all the feedback they've given us.

What open source components does Cloud Firestore use?

Can you point towards papers that describe the algorithms and data structures that Firestore uses? What infrastructure does it rely on to perform reliably?

This isn't true, but Cloud Spanner is awesome too!

It's probably better said that Cloud Firestore and Cloud Spanner share common infrastructure.

For example, Cloud Firestore uses TrueTime to provide similarly strong consistency guarantees to Cloud Spanner (see https://cloud.google.com/spanner/docs/true-time-external-con...)

I'm not aware of any whitepapers describing the listen feature that provides real-time updates.

The backend is almost entirely custom, though many common low-level open-source components are used pervasively. For example, for components written in C++ we use Abseil, Google Test, etc.

The SDKs are all open source and make use of other open source components:

  * Android: https://github.com/firebase/firebase-android-sdk/tree/master/firebase-firestore
  * iOS: https://github.com/firebase/firebase-ios-sdk/tree/master/Firestore
  * Web: https://github.com/firebase/firebase-js-sdk/tree/master/packages/firestore

For example, the iOS SDK uses LevelDB for local key-value storage and gRPC for communication with the backend.

You can find language specific libraries for many popular languages under the GoogleAPIs org on GitHub. To install, go to the defacto package manager for the specific language.

* .NET: https://github.com/googleapis/google-cloud-dotnet/tree/maste...

* Go: https://github.com/googleapis/google-cloud-go/tree/master/fi...

* Java: https://github.com/googleapis/google-cloud-java/tree/master/...

* Node.js: https://github.com/googleapis/nodejs-firestore

* PHP: https://github.com/googleapis/google-cloud-php-firestore

* Python: https://github.com/googleapis/google-cloud-python/tree/maste...

* Ruby: https://github.com/googleapis/google-cloud-ruby/tree/master/...

I'm interested in learning about Firestore architecture and how it solves difficult problems to understand and trust it. I'm not interested in client libraries.

Sorry I misunderstood. I was simply sharing additional open source components that are part of the Firestore ecosystem. Best of luck in your search.

What's the status of Firestore under the BAA along with the other Fire* services?

Cloud Firestore supports HIPAA compliance, we are working to update the documentation here: https://cloud.google.com/security/compliance/hipaa/#covered-...

I didn't know this -- this is very very cool! I remember following an old google group of people who were waiting for that support.

Thanks for posting this! I'm surprised it wasn't mentioned in the OP.

I love Firestore, using it pay per go to push a lot of bytes, even some logs to see them real-time sync in browser and it scales really well and it is cheap.

"Serverless", hey?

Yes, it runs on racks of Google Pixel devices rather than servers. (I jest)

I like cloud firestore and firebase platform very much. It would be nice to have more query features. A simple "!=" is not possible.

Also an automatic backup would be nice (and can't be that hard to offer?).

We're looking at more comprehensive backup solutions, but in the meantime we have a fully managed export system you can schedule (https://cloud.google.com/firestore/docs/manage-data/export-i...) with the added benefit of this exports being importable into BigQuery for ad-hoc analytics (https://cloud.google.com/bigquery/docs/loading-data-cloud-fi...)

I've used Cloud Firestore extensively in some of my personal projects. I also have lots of feedback and suggestions I'd love to share with you if interested. Is there still a way to participate in this program or get in touch with you?

Our discussion group has a lot of the team, including me, on it: https://groups.google.com/forum/#!topic/google-cloud-firesto...

Thanks for the great product. Will support for geo queries be added anytime soon?

Thanks! It's something we're discussing, but won't be "soon".

Waiting for geoqueries also. Would make a bunch of features in a few of my apps so much easier.

I'm a big fan of firestore and live in Colorado. Are there any community events that you host or recommend that would allow me to talk to you, other people at Google, or other firestore community members?

I'll talk to our Developer Relations team in Colorado to follow up!

Hi Dan,

any chances you and your team are working on a GraphQL access layer for the Firestore?

It feels like nested queries(using refs for relations), basic mutations and (especially) graphql subscriptions would be a great match with Firestore and a big improvement in dev experience, imho.

We aren't, but we know customers to create GraphQL access layers themselves.

The point of having a GraphQL layer is not having to make it ourselves.

https://aws.amazon.com/appsync/

https://hasura.io/

I have created a GraphQL layer over the top of Firebase and while it was pretty easy to do so using Cloud Functions, I agree that it would be great if Firebase offered this out of the box without needing to do all of the work you currently have to, to use it.

Hi.

As a student developing a system for a friendly NGO I must say that having angularfire integrate with Angular made this an instapick because I wanted to get started asap.

The issue now is that it really feels like the docs need more examples. I'd take 4-6 hours to add a new feature and I'd spend them reading the docs. Sometimes I'm not sure if I don't get something or it's a missing feature.

I guess the community is still growing but it's really hard to find answers to some simple problems.

angularfire has an inactive gitter and there isn't one for firebase specifically. It would be really amazing if firebase devs could take some time to answer the community on gitter or something, at least at the beginning before there's more people knowing the answers.

EDIT: Found your link to the google group. I guess I'll ask there then :)

Thanks for the feedback. Feel free to send any feedback on missing examples/tutorials that would have made things easier to our discussion group (https://groups.google.com/forum/#!forum/google-cloud-firesto...). Our tech writers see the feedback and use it to help plan their work.

AngularFire lead here, I'd recommend you check out our GDE Jeff Delaney's community here https://angularfirebase.com for AngularFire related content. He has tutorials, lessons, and maintains a very active Slack Channel.

There's also the Firebase community Slack https://firebase.community/ and firebase-talk is our supported community channel for all things Firebase https://groups.google.com/forum/#!forum/firebase-talk

Hi, I’ve enjoyed using Firestore so far, but one aspect I find very limiting is the fact that you are limited to one Firestore database per Google Cloud project. Are there any plans to change this? Allowing multiple Firestore databases would make setting up staging and testing environments much easier.

Usually you should use separate projects for this. Dev, staging, production projects - separated permissions, rate limits and costs

Hi there. :) I'm an engineer on the Cloud Firestore team. I can't speak to if or when we'll be able to remove this restriction, but please know that the folks working on this (like me) are very aware of what a pain in the butt this is. I'm sorry that I can't give you a more satisfying answer, but I hope we'll be able to change this soon. Thanks for being a customer!

Is it a BaaS? Where does it fit in in the landscape?

Why can’t I use it in the same app as cloud data store to build new functionality on the modern api?

Why isn't there a hosted, easy to use relational version of Firestore? I'm using Firestore right now and it's absolutely frustrating in the lack of relations, the lack of accumulation queries (COUNT, MAX, MIN) and in general the limitations of storing what's essentially arbitrary JSON. How is it that to count the number of entries you either need to build in a counter on your own (which can cause race conditions because counters are hard) or manually go through the entire collection and count?

Another option would be a typed JSON db, essentially you could store JSON that corresponds to typed structs a la serde. Would't solve a lot of my problems, but at least I'd have some built in validation.

Because relational databases are much harder to scale. There are many people working on solving this problem.

At what level? Because I know StackExchange runs off of relational databases and they get a solid amount of traffic (with some Redis, but the foundation is relational). Even if relational doesn't scale for Facebook or whatever, if I'm making a piddly little chrome extension or a small website, a simple relational database will scale just fine.

Firebase is designed around syncing, and I imagine that has something to do with it.

If you want a relational database for just one user's data, you could read a SQLite data file over the network when your user opens the app, perform SQL queries over the data in memory, then write the entire database file back across the network when the user hits a "save" button.

But you probably don't want a "save" button, you just want the user's state to be constantly updating back to the server.

And you probably don't want to write back the entire data file every time you save, just the parts that have changed.

So Firebase is optimized to solve these problems, and I bet the data structures they use are not also optimized for supporting all kinds of relational queries.

It's important to note that StackExchange is one of very few examples of successful vertical scaling, at least in the post-mainframe world.

If you want a relational DB as a cloud service, it sounds like you want Cloud SQL (as far as GCP is concerned).

(Disclaimer: I work on GCP.)

Cloud Spanner is also relational. :)

(Fellow GCPer.)

I only know of FaunaDB which is not really relational.

Do you have any other suggestions?

If you want a scalable relational database, that's Cloud Spanner.

Well, I'm actually more interested in serverless databases.

An earlier commenter noted that one of the features of Firestore is user-level security, i.e. clients can only access their own data.

This makes sense for many apps in a document-based/NoSQL model, and may let you avoid any need for your own back end.

But it makes less sense when you start introducing relations and data models become more complex.

e.g. pretty soon we'll need to join to a lookup table. That presumably means that that lookup table needs to somehow be opened up to all users, while for other tables the user can only access their own data. Starts getting complex real fast.

... Spanner? It's by the same company.

Looking at pricing, it's a little out of my budget. But thanks for the suggestion!

Firestore is literally built on top of spanner. So yeah, that's the answer. It's definitely pricey at first glance but it's a fixed cost based on nodes, compared to say datastore where you pay for storage+reads+writes. Different use cases.

> Firestore is literally built on top of spanner.

Do you have a source for this? Thanks in advance.

If Cloud Spanner is too big, then you'll almost certainly be well served by Cloud SQL (fully managed MySQL and PostgreSQL): http://cloud.google.com/sql/

Check FaunaDB https://fauna.com/

While the move seems to be towards NewSQL databases (Spanner/CockroachDB), the answer to relations in NoSQL is to model the data differently. This can generally resolve a lot of the problems inherent in the need for relations.

This might not be useful advice for everyone, but I can highly recommend this youtube video on DynamoDB data modelling: https://www.youtube.com/watch?v=HaEPXoXVf2k

Having heavily used Firebase Realtime Database (firestore's ancestor), I think I will approach this one very carefully. Firebase Realtime Database was a nightmare with frequent downtime, sometimes minutes, occasionally an hour. Also almost weekly, all clients sometimes wouldn't get notified of document changes which was crippling for our app.

I've been using cloud firestore for 6 months now and haven't had an issue with it. Designing around their rules system has been annoying, and the lack of backups bit me once pretty hard (I built https://firesafe.app as a result), but other than that it's been great.

I think one of the requirements for Firestore going GA was getting a b̶a̶c̶k̶u̶p̶ managed export system:

https://firebase.google.com/docs/firestore/manage-data/expor...

Edit: As wsh91 says below, you are still responsible for the scheduling.

(I work for GCP)

right, but didn't see anything mentioned about backup in the link. or am i missing anything?

An export is one component of a backup solution. :) Scheduling is another (https://firebase.google.com/docs/firestore/solutions/schedul...). (Checking restores is critical, too; that's also feasible [managed import into a new cloud project] but requires a bit more legwork.)

(I work on Cloud Firestore.)

FWIW, I'd love it if you made my app unnecessary. You can consider my site a list of feature requests :)

Oh wow!

I am just using this https://github.com/ChristianAlexander/FirestoreRestore with cloud functions & cloud storage.

Hi EZ-E - completely understand the concern. Cloud Firestore's architecture is very different and engineered to be an extremely highly available system backed by a 5 nines SLA. It has no scheduled downtime either.

We once had a 13 hours downtime on our main product. It wasn't a fun day.

We decided to move out of the Firebase RTDB because not only we weren't happy about it but it was unreliable.

We still use it's static hosting and cloud functions.

Firebase Realtime was basically a Mysql instance which Google ended up with when they bought the Firebase ecosystem. Not scaleable at all. You see hints about this in some places when they talk about using it in production. I am sure some Google engineer went nuts when he saw what they have to support now.

Firestore is a complete engineered Google solution based on the Google Datastore but with added features from recent years (spanner, cloud functions). This is very obvious if you compare the limits between Datastore and Firestore.

IMHO they are not really comparable.

I still haven't seen a great reason to go NoSQL over, say, Postgres. I'll think about a good application but then realize that it'll be a PITA to do something slightly different than what I first imagined.

What are good reasons to use NoSQL over SQL?

Usually the bigger reasons are easier scaling. By taking a different approach to data, it's often easier to scale across data centers etc. The second being the shape of your data. If most requests are simple key/value queries, a much simpler model can work better, more so if you can keep all your data for a given query together (document dbs in particular), this can have performance benefits over a typically normalized database.

Another is when you need read and more specifically write performance that a single system cannot keep up with. When you hit these boundaries, it gets interesting. Sometimes it's just easier to design for such a system up front.

If it's an internal application SQL first is probably fine, if it's SaaS you may want to look at alternatives.

I'd love to use Postgres, but Firebase is really nice in that it provides first class libraries that abstract away the requests and querying, to the point where querying Firestore just feels like calling an async function on your front end. Plus, you don't even have to think about deploying. There's something really great (and risky) about having an always deployed, always ready backend. Also basic user auth is really easy.

If you're doing a long term project, then Postgres makes sense since the deployment/setup costs are one time. But for short projects Firebase is very nice.

But there are managed Postgres services in almost all cloud providers.

"SQL vs NoSQL" is mostly just buzzy clickbait type arguments.

My understanding is for Firestore specifically, its a document store with some conflict resolution semantics that Postgres does not provide.

All other things being equal, I wouldn't go NoSQL over Postgres.

What makes Firestore interesting to me (not used it in production) is that you can avoid the need for a backend completely. Your normal architecture is Client <-REST-> Backend <-> DB. You can avoid all the deployment and development in the backend by doing the work in the client.

"More features coming soon. We're working on adding some of the most requested features to Cloud Firestore from our developer community, such as querying for documents across collections"

Looking forward to it. If this comes in, pretty much it removes the need for creating top level collections or am I missing anything?

I've been using Firestore in a React app and I love how it has simplified everything. There's no need to have redux or even an in-memory store in the "context". I just wrap my components with a withFSQuery helper and it automatically updates even when the state changes "server-side".

Would you have time to elaborate? I’m just starting to set up a React-Firestore app but I’m a bit of a n00b.

I’m using re-base [0] to synchronise Firestore to my local state. Do I not need to be?

Any pointers gratefully received, thanks.

[0]: https://github.com/tylermcginnis/re-base

I didn't use any library. Seemed easy enough to just create a few helpers. I rely on Firebase completely for the data. Since Firebase already keeps a local client cache as well as an offline cache, I didn't see any need to maintain separate state in React -- outside of the HOC components. They subscribe automatically to the data when the component mounts and unsubscribe when the component is destroyed. Any changes, whether from within the app, from the server, or from another instance of the app; have changes synced everywhere for free.

The only local state I have at the app level is state that doesn't get stored in the DB.

Here's the helper HOC I created for it:

https://github.com/brennancheung/wowzrs/blob/master/src/app/...

And an example of it being used as a render prop:

https://github.com/brennancheung/wowzrs/blob/master/src/app/...

And another as an HOC:

https://github.com/brennancheung/wowzrs/blob/master/src/app/...

And here's a mutation:

https://github.com/brennancheung/wowzrs/blob/master/src/app/...

The HOC's make it so you can create just plain stateless functional components and not have to worry about componentDidMount.

We have been using it for over a year and this is good news. Hopefully a good product will move to a great one.

A months ago I published a write-up of building our our cloud sync in Polar:

https://getpolarized.io/2019/01/03/building-cloud-sync-on-go...

Polar is basically a document management and annotation platform. You put all your reading in it and maintain it long term along with annotations, highlights, etc.

Firestore is really nice in that you can target multiple-platforms pretty easily. There are SDKs for basically every platform.

It's definitely not perfect but I'm pretty happy with the decision.

Will there ever be support for aggregate queries? Or queries for non-existent values?

I have a rather large dataset that's tough to scan over and I find the functions/transactions to build my own aggregates not entirely accurate enough since a document has rate limits for writes.

It's a point of internal discussion on scalable ways to achieve this, but nothing we can promise. We definitely see the need for it.

I totally understand its a complex issue. Is there someway I could subscribe to updates regarding this subject for the future (even if it's 2 years away I would enjoy testing any potential developments here)

Hi there! One thing you might consider is using a function to replicate your Firestore data to BigQuery, depending on the value the latter would add. (Very fast OLAP; I'm a big fan.)

(Cloud Firestore eng. here.)

The issue we had with cloud Firestore is the inability to _really_ query documents through the Firebase console. It’s very basic and there is no third-party tooling available yet.

To run any kind of specific query you’d need to handle that application side. Just something to consider.

This service feels pretty shoehorned into the rest of GCP. There are little connectors everywhere for Firestore specifically, and they don't fit in cleanly with the rest of GCP offerings. Why is Google so bullish on this thing? I must not be seeing the appeal.

(I work for GCP)

Where do you feel it is shoehorned in? Firestore is easy to access from App Engine and Cloud Functions (and GKE/GCE as well) using the built-in service accounts, it has managed export into BigQuery. Would love to know where you found friction when using it with other GCP services.

> Why is Google so bullish on this thing?

While I can't speak for Google as a whole, I'm personally bullish on Firestore for a few reasons:

1) Gives you the same "client-side only" model that Firebase popularized. This makes creating applications much faster as you remove the whole server side component and basically all your ops work.

2) True "serverless" pricing and ops. Because you only pay for reads/writes and not instance time, your costs scale linearly with your app's usage. And you don't have to worry about sharding or other operationally complicated things to scale your database as you grow. The caveat here is you have to structure your data and queries well, otherwise your costs will skyrocket [0].

3) Gives you the same "server side" functionality as Cloud Datastore (which Firestore is basically the next generation of [1]). This means you can use Firestore in place of a traditional NoSQL database like MongoDB.

4) Strong consistency. One of the biggest problem with Cloud Datastore (and most NoSQL databases) is eventual consistency. Datastore worked around this with "Entity Groups" which in my opinion were super clunky to work with and very limiting. Firestory is strongly consistent so you don't have to worry about this even at scale, which is super nice.

5) GCF Triggers. The fact that you can trigger a cloud function when something is written to the database is very powerful, it's basically like a traditional database trigger or stored procedure but you can do anything.

The biggest feature gap for Firestore is the querying abilities. While it is way better than the original Firebase DB, it's nowhere close to a relational database or something like Cloud Spanner. The team is working on it though.

The "real time" stuff is interesting but not really super relevant to the things I like to build.

[0] https://hackernoon.com/how-we-spent-30k-usd-in-firebase-in-l... [1] https://cloud.google.com/datastore/docs/upgrade-to-firestore

I mentioned it in a comment above, but one way in which it feels shoehorned in is the fact that you can only have one Firestore per Google Cloud project. Having only one Firestore per project makes setting up test or staging environments a hassle.

Oh yeah, this fact is definetly my least favorite part about Firestore. Also the fact you can't change the location once it is created.

I don't know what the roadmap is for removing these restrictions (or even if it is on the roadmap) but I'll update this post if I learn more.

In GA? Like the State?

General Availability. It has been promoted from a Beta product.

If I remember correctly from last time it means "general availability" which is some internal corporate lingo that most people won't understand.

It's not internal corporate lingo, it's how business that produce products plan the lifecycle of their products, and while it's commonly used in software, it's a very standard term across verticals.

curious: how big is the Firestore team? what about the team for the Firebase products as a whole?

Will there be read-only replicas in other regions available in the future?

It's technically possible for us to build that, but we're focusing on the most common requests at this time. If anyone has that please let your account reps know (they feed us lots of great feedback for prioritization), or start a discussion on our Google Group: https://groups.google.com/forum/#!forum/google-cloud-firesto...

Yes, this is what we like about RDS/DynamoDB/CosmosDB. Being able to run low-latency replicas (bonus if they accept writes) is important for distributed apps, but the current multi-region deployment adds too much overhead.

Global replication simplifies a lot of things.

Any notable companies/products known to use Firestore in production?

https://getpolarized.io/

We are... We're not massive yet but we have about 250 users who've updated about 20GB since our launch. Still really early as we haven't had a massive launch yet.

We're still soft launching.

We listed a sample in the blog post, including The Telegraph, The New York Times, Skip Scooters, Now IMS, and Hawkin Dynamics

The blog post has a lot of customer testimonials :)

So next up we'll finally get an API for configuring firebase right?

The Firebase Management API (https://firebase.google.com/docs/projects/api/reference/rest...) was announced at the Firebase Summit last October, and will expand to provide more configuration over time.

I want to use Firestore in Unity. When???

any implications for regular old Firebase?

None, still a fully supported option for users.

I find the limitations of their query language interesting, no way to test for undefined properties in a collection (the client libraries just give up, and I don't see a workaround); and no way to match on inequality rather than equality. This exists in some other odd database engines, but it's frustrating in general use scenarios (where, in this case, I don't exactly see how performance is a limiting factor).

Be careful when storing data. Flat earth data could be deleted :) it violates community guidelines lol

Reply on news.ycombinator.com