Comment by jamest | Hacker News Reader

Comment by jamest | Hacker News Reader

Parent Full thread

[Firebase founder] This new database has been in the works for 2.5 years, since shortly after we joined Google. It was developed in close collaboration with the Cloud Datastore[1] team, and uses Google’s core database infrastructure.

We built it because we know it can be challenging to build complex apps with our original database -- Firebase Realtime Database -- where we optimized for ease-of-use & real-time sync over querying functionality. For more info, see this comparison between the two[2].

This is a open beta launch, so the product has limitations[3] you should be aware of. We’ll be working to remove/raise these before General Availability.

A note on naming: Firebase launched in 2012[4] with a single product, the original database. As we added products (Firebase Auth / Firebase Hosting / etc), we began calling the original product ‘Firebase Realtime Database’, or RTDB for short. If you haven’t followed us lately, we’ve grown to become Google’s app platform, and now have 16 products to help you build/grow apps.

We’re grateful for the support HN has given us over the years and we hope you enjoy Cloud Firestore!

[1] https://cloud.google.com/datastore/ [2] https://firebase.google.com/docs/firestore/rtdb-vs-firestore [3] https://firebase.google.com/docs/firestore/quotas [4] https://news.ycombinator.com/item?id=3832877

How goes the work to improve customer service and support?

Thinking about https://news.ycombinator.com/item?id=14359801, what improvements have you made in the last 139 days that would make us want to build on this?

For their own sake, I hope the team gives this question attention and respect.

What sets Amazon apart from many other companies is their reputation for relentless customer service. Customers pay close attention to the quality of answer or lack of answers to questions like this one.

Every cloud company that asks us to invest our time in their proprietary API is asking us to trust them. If you violate one vocal customer's trust, we're going to notice. If you don't make things right, we're going to notice. Even before I read the article, I first read the comments to gauge the reception because I care more about the perception of trust before I even consider using something like this.

Customers don't always stop trusting companies for fair or valid reasons, but that doesn't matter. Trust is much more of a visceral thing than a cerebral thing.

So here's my question to other HN readers. What would Google need to do to improve its reputation for customer service? What would they need to do to make that commitment resonate with you on a visceral level?

For me it's pretty easy. I want to start hearing and seeing positive anecdotes about real time communication with Google Cloud support. That is to say, I want to hear people start saying things like "I rang them, and everything was sorted out" ideally with a platitude attached like "it was actually a pretty good experience".

I hear this about AWS all the time, and I've even experienced it myself. One client some time ago had an AWS novice who was confident they could handle setting up the auto-scale groups. They made a small mistake, which lead to the scaling trigger being permanently on, and it auto scaling to 1024 ec2 nodes within the first half hour. Immediately after deploying a fix for the scaling trigger, I phone AWS to see if there was anything that could be done about the cost of this accident, and I had a small monthly credit to compensate, and a respectable discount on the next appropriate AWS training course so that the novice could learn more and hopefully avoid future mistakes.

THAT is the kind of story I want to start hearing about Google Cloud. "I used it and did not have problems", and "It works and was cheaper than AWS" is just not enough.

So my reading here is that the main issue is Firebase was undercharging customers, due to a bug.

They didn't go back retroactively and ask for back payments, but just started charging them accurately going forward.

Of course, that bill caused shock for some people, as they suddenly realised, "Oh s*it, I'm being charged accurately now, my bills are huge!!!"

In this customer's case, apparently they were also doing something bad with TLS tickets and not setting keep-alives - basically causing them to spam Firebase with connections:

https://news.ycombinator.com/item?id=14357270

Of course, there's the other issue of a communication breakdown which they said they're working on.

(Some quick context for those not aware of the initial incident: In May we fixed two bugs in our pricing pipeline that caused some users to see price increases.

Bug 1: we under-reported bandwidth (in particular SSL overhead)

Bug 2: we were not enforcing quotas for all accounts.

For most users, the fixes had little-to-no impact. For a few users who were using the Realtime Database with large volumes of small reads and writes, the impact was large. You could mitigate this impact by updating your client code, but unfortunately a user who had shipped code to their IoT devices couldn’t. This user was also simultaneously forced to upgrade to the Blaze pay-as-you-go plan due to quota enforcement on our $25/mo Flame plan. These combined resulted into a large billing increase for this user. We weren’t quick enough to provide this user with credits due to poor internal communication).

To address these we have (1) worked to make billing more transparent on Realtime Database and (2) are working on improving support.

1a. We rewrote our documentation to add more detail on billing mechanics and how to optimize bandwidth (https://firebase.google.com/docs/database/usage/billing).

1b. We rewrote the documentation for our profiler tool which was confusing to many developers (https://firebase.google.com/docs/database/usage/profile).

1c. We now have better alerting for if/when we find errors in our codebase that can impact their bill (up or down).

1d. We will soon be releasing (spoiler alert) a new monitoring API to let developers directly analyze their database billing and performance data.

2a. We raised the quota on free technical questions from 5 => 10. Questions on accounts/billing/bug reports are still unlimited

2b. We worked to increase Support CSAT. It is up by 15% since the billing issue in May.

Finally, the new database we’re launching today, Cloud Firestore, has daily budgets. You can use these to set exactly how much you’re willing to spend per day (more here: https://firebase.google.com/docs/firestore/usage#limits) We’ve also got extensive pricing docs: https://firebase.google.com/docs/firestore/pricing

I hope this answers your question!

> You could mitigate this impact by updating your client code

Where can I read about this?

You can find more on this in the "understanding billing docs" : https://firebase.google.com/docs/database/usage/billing

Look for the optimize billing section

How does customer service work with technical services?

There is very little time for developers of a system to use on support, so how do you go about building the baby padding around them? I would imagine at least 97.5% of support requests are a problem on customers end, so just how much support staff would you need for a global service like this?

Just read through your link #2 and was very impressed. Firestore addresses every single concern I had with RDTB and more and I'm looking forward to using it in my side projects.

Thank you!

That's great to hear! Our design for Cloud Firestore was based on years of developer feedback. Glad we hit the mark for you : )

What's your story on ACID, particularly on transactions?

Transactions: If I want to insert 2 documents, but zero if either fails, what does that look like with Firestore?

And if I store a document and run a query milliseconds later, will the query include document I just stored? Or are queries eventually consistent?

Cloud Firestore has support for transactions: https://firebase.google.com/docs/firestore/manage-data/updat...

So yes, if any of the operations in the transaction fails (and cannot be retried) the whole transaction will fail. It's atomic as you'd expect. If you've ever used transactions in Firebase Realtime Database you'll be happy to know that the transactions in Cloud Firestore are much easier to use since they don't require selecting a common data parent.

Queries are always strongly consistent. If you do a write-read sequence you'll see the value of the latest write.

How do you deal with consistency in the offline case? E.g. If I make a bunch of edits while disconnected, and others have made edits while connected, how are the conflicts resolved?

This works similar to the Realtime Database in that it's last-write wins (where in the offline case, "last" is the last person to come back online and send their write to the backend). This model is very easy for developers to understand and directly solves many use cases, especially since we allow very granular writes which reduces the risk of conflicts. But for more complex use cases, you can get clever and implement things like OT conflict resolution as a layer on top of last-write wins, e.g. similar to how we implemented collaborative editing with www.firepad.io on the realtime database.

PS: Hi Chris! :-)

Hey Michael! Congrats on the launch :)

Providing a one-size-fits-all solution here is probably impossible, but it seems like it would be nice to provide some mechanism to be notified that you're making edits based on stale information. If such a mechanism existed, it would be easy to add a bunch of canned merge strategies. In doing so you can probably teach people a little bit about the pitfalls they're likely to run into (these sorts of bugs are insanely difficult to track down), while not really making them do much work.

The approach we've taken in Eve is that we can't solve all these problems for you, but we can at least let you know that things can go sideways and prompt you to make a deliberate decision about what should happen. It's amazing how helpful that ends up being.

Thanks for the feedback. I think you're right and we're interested in exploring what we can do to help people more in the future. One of the really nice things about Cloud Firestore is that documents are versioned with timestamps in such a way that we could definitely detect and expose conflicts and let you decide how to deal with them... It's mostly a matter of identifying the common use cases and then figuring out the right API to make them possible without going too far into the deep end of conflict resolution.

It looks to me like Firestore's API doesn't include a "default" way to upload user edits to documents. Conflict detection is possible using the transactions - https://cloud.google.com/firestore/docs/manage-data/update-d... - you can do something like HTTP's PUT If-Unmodified-Since (or PUT If-Match).

Good point. read-modify-write transactions are a good way to detect conflicts and get a chance to handle them, but they're unfortunately limited to while the client is online. If the client is offline, the transaction will fail so they're not useful for general conflict resolution. This was an intentional decision because there's not a straightforward way to preserve the intent of the transaction across app restarts. But there may be options for adding some sort of conflict resolution strategy in the future that leverages the same underlying primitives that transactions use today.

> OT conflict resolution as a layer on top of last-write wins

Can you link to somewhere where this layering is explained?

The www.firepad.io site has documentation on how to use the editor, but I'm interested in how "OT on top of last-write wins" is achieved.

Awesome to see this. I couldn't get behind the RTDB structure, but Cloud Firestore I will definitely use!

Woohoo! Is this an official parse endorsement? (Hi Fosco )

Hey Thomas :) I speak only for myself, Firebase just got way better.

I'm glad. Hoping you like cloud functions too!

Very exciting. I found this to be the one "missing piece" keeping me from being able to build a whole app on Firebase when checking it out earlier this year. One question though, will this support BLOBs like GridFS?

You can store Blob data, but the maximum size of a single document is still 1MB. If you need to store larger blobs check out Cloud Storage for Firebase.

Could you share some of the reasoning behind arriving at a 1mb limit (why not 10mb or 512kb for example)?

Hard to explain externally, but it relates to RAM size allocation + throughput in the underlying distributed database system.

Regarding offline mode. How is sync done? Is this OT based? How are conflicts resolved?

Good question, and to answer this well we should probably do a blog post or something. In the meantime you could dig into the code since the clients are all open source. :-)

But basically, sync is split into two halves: writes and listens. Clients store pending writes locally until they're flushed to the backend (which could be a long time if the app is running offline). While online, listen results are streamed from the backend and persisted in a local client cache so that the results will also be visible while offline (and any pending writes are merged into this offline view). When a client comes back online, it flushes its pending writes to the backend which are executed in a last-write-wins manner (see my answer above to ibdknox for more details on this). To resume listens, the client can use a "resume token" which allows the backend to quickly get the client back up-to-date without needing to re-send already retrieved results (there are some nuances here depending on how old the resume token is, etc.).

Thank you!

Good work guys, sounds like good progress!

I'm struggling to see what this is giving me that I didn't have with RTDB though

The queries seem to be doing what orderByChild equalTo startAt endAt limitToFirst and limitToLast were already allowing.

Is there mostly a performance gain, or am I missing something that I can do now that I couldn't do before?

Cheers

I'll answer myself here for anyone else trying to figure out what we can do with queries now:

---

We can now apply a filter and sort in the same command

We can chain filters, though they can only apply to one field if they specify a range

We can make a query that will only return documents, not the subfield data (i.e. what a lot of us were doing with the deprecated REST interface shallow=1)

Queries in large data sets will be faster I guess?

Things it'd still be nice to see -

References that allow you to pull down related documents with a single query

OR queries

---

It's been a bit of rollercoaster seeing this announcement..

"Oh wow we can do proper queries now! Oh wait no, the queries don't seem to allow more than what we had before.. oh wait there are some improvements, it's a bit better now.. oh the shallow querying will be super handy! and they do geopoints.. but don't appear to have any way of searching for radius.. but they say they will soon"

Anyway, good work all the same :)

Can you provide anything to reassure potential users that this will still exist in five or ten years time?

The Firebase Realtime Database has been around for ~5 years (and is still fully supported after this launch).

Cloud Datastore has been around in some form (started as App Engine Datastore) since 2008.

Cloud Firestore was a massive multi-year joint effort between Firebase and Google Cloud. Google is investing heavily in both, and this is a big deal for our teams.

Perhaps you could consider persuading the powers that be to make a loud public commitment.

After all, if they are truly confident, there should be little cost in doing so.

We announced the product, that's a loud public commitment. Once it reaches general availability, it will be covered by the Cloud deprecation policy requiring a minimum of one year notice for deprecation.

I'm not sure what other guarantees would even make sense to offer. If anything, I'd look at this announcement as ongoing proof in the magnitude of investment Google is making in Firebase and Cloud.

I'd guess a forward commitment, like a LTS version? That would probably have an adverse effect though, since you'd be the only provider that goes (e.g.) "We will support this product for at least three years from now", implying (to people making Decisions) you'd pull the plug after three years.

Maybe publish a long term (5+ year) plan / roadmap? idk.

Yeah, unfortunately this is pretty much an unsolvable problem.

Roadmaps are subject to change and even more subject to be delayed, publishing them tends to disappoint more than reassure. If we gave a forward commitment the questions would just be "why not longer?" or "what happens in X + 1 years?".

All we can do is say what I'm saying now: we stand behind this product 100%, we think it solves real problems for developers, and we really hope people will try it out and find it useful.

I get the doubt, truly I do. But Google's incentives are clearly aligned with Cloud Firestore's success: if you folks use it and grow your app to be successful, we make money. If you use it and really like it, you're more likely to use Firebase and Cloud's other products, which will make us even more money.

>if you folks use it and grow your app to be successful, we make money. If you use it and really like it, you're more likely to use Firebase and Cloud's other products,

the issue is the inverse is also true: If you folks don't use it, we don't make money, and we deploy these resources elsewhere. See Parse

While I know this is a beta and you’re solving a difficult technical problem, some of the limitations are concernig. With a system like dynamo, you can in most cases pay your way out of bottlenecks.

It would be nice to develop a way in Firestore for users to be able to pay for the number of shards assigned to their workload to reduce relatively artificial limits such as write limits within collections and index update rate.

That way, developers wouldn’t have to plan to leave the platform if their app is successful.

The enhanced query capabilities are very much appreciated. I hope full-text search is in the works, since adding a 3rd party FTS engine adds quite a bit of complexity.

Do you have a rough timeline on beta => General Availability?

Nothing we can share at this time. Really looking forward to the feedback we get now we're in beta as this will feed into our GA plans.

glad you mentioned Cloud Datastore. It seems that Firestore supersedes Datastore and Firebase. Is that correct? Any comparisons of Firestore vs Datastore?

It does not "supersede" Datastore or Firebase Realtime Database, neither of those products are going anywhere.

This is a totally new product that builds on what we learned from those two products, sharing some of the best features of each and bringing in some totally new things as well.

All three are (in my opinion) great choices for building a new app today so make sure you evaluate all options.

Do you see this as an equivalent to Dynamodb ?

Very timely ;)

DynamoDB does not come with offline sync.

Reply on news.ycombinator.com