Hacker News Reader

Comment by jamest | Hacker News Reader

[other Firebase founder] It was painful to read the article[1] this morning, especially since I was one of the people responsible for dropping the ball on getting Home Automation the credit to cover the overage a few weeks ago. We're working with the founder to make sure he's in a better spot. If you have similarly serious issues, my email is: james@firebase.com

To address a couple of points that have been raised:

1. We're aware that as we've integrated with Google our support response time & quality has decreased. I'm working with our team to do better.

2. We know better querying and web offline are needed for the Realtime Database, stay tuned.

Finally, I hope you enjoy all the new features that launched today! (https://firebase.googleblog.com/2017/05/whats-new-from-fireb...) Leave a comment if you have thoughts/comments.

[1] https://news.ycombinator.com/item?id=14356409

View on HN

>1. We're aware that as we've integrated with Google our support response time & quality has decreased. I'm working with our team to do better.

Can you though? I've yet to see any good google support for any software product. How much leeway do you actually have to change the support culture of a company that doesn't care about support?

(not affiliated) we are using compute engine and response time are great. Nothing we can complain about.

Both Fabric & Firebase (pre-acqusition) had great support cultures. I think we have a good shot at improving how Google approaches support for Firebase.

The proof will be in the results. Hopefully we can share those in the future.

I know I'm late to the party, but I really hope that things get back on track soon.

When I learned Google had acquired Firebase, I was really disappointed because I knew that it would be another instance where a great product or service was crippled by Google's approach.

I came across Firebase fairly early on, and I absolutely loved it. Documentation was solid, the examples were all interesting and straightforward, and support was always great. Now, documentation and site navigation are both downright painful to deal with. I've recently transitioned multiple projects to Deepstream.

Personally, I'm not confident enough in the direction things have headed to commit to Firebase beyond trivial projects. I got in on Fabric in Jan 2015, and again, it was a great experience. I just feel like Google's lead to a decrease in focus on the core of what makes Firebase so appealing.

Like I said, I'm a long time Firebase guy, and I really want to see things improve. At the very least, can I give up my first born for a documentation overhaul? I swear it wasn't always this frustrating and cluttered.

It's good that you're able to fix this sort of thing when it's on the front page of HN, but Google's lack of support is legendary, and I don't see why people should expect anything if they aren't making the news like that. My own experience with Google Glass DevRel was about as bad, except no one made any attempt to set things right, or even apologize.

To my discredit I enjoyed dumping on Google earlier today with the Firebase support issue article.

That being said this was a great response and I appreciate it. Also my first React Native app used Firebase and I have fond memories of setting that up :)

I like that item #1 was very direct...essentially: "look the support got worse and it's not good but we're working on it".

SIDE NOTE: I don't get how a lot of people today don't understand that using vague language in apologies doesn't help with apologies. It definitely used to, but then everyone started doing it, and now the cool / rare thing to do is to be direct. Maybe the pendulum will swing back someday. Until then thank you Mr.Google/Firebase/James person.

I enjoyed dumping on Google earlier too. I love firebase, but I've talked two companies I've worked with out of using it for new, mission critical projects. The earlier article made me feel completely vindicated in my recommendations.

I simply have no trust that:

- Google won't randomly cancel firebase (wave, reader, etc)

- Google won't randomly, suddenly jack up the prices for firebase, leaving users in the lurch (appengine)

- When something mission critical breaks or changes, I'll have any way of contacting someone who cares, no matter how much my company is paying for the service. (Google's support forums are a pit of sorrow)

I was reading an article the other day talking about personhood[1]. It makes the point that part of being a member of society is the idea of standing. That is, if you break your word there has to be a way for you to lose out as a result. If there isn't, its impossible for anyone to enter into an agreement with you because you can't be trusted. Forming an agreement requires both sides to have skin in the game.

I don't genuinely believe that google has skin in the game when it comes to cloud services for small-to-medium customers. Its just too easy for them to drop the ball, stop answering emails and leave customers in the lurch. They've done it again and again. I'd say its the default way Google operates.

Thomas Schelling from The Strategy of Conflict:

> Among the legal privileges of corporations, two that are mentioned in textbooks are the right to sue and the "right" to be sued. Who wants to be sued! But the right to be sued is the power to make a promise: to borrow money, to enter a contract, to do business with someone who might be damaged. If suit does arise, the "right" seems a liability in retrospect; beforehand it was a prerequisite to doing business.

Please, firebase. Please be different. But know that thats the reputation you're fighting against. Thats the reputation you've inherited by joining Google. But for now I'm standing by my earlier recommendations.

[1] http://www.meltingasphalt.com/personhood-a-game-for-two-or-m...

As I said two years ago¹, Google is too large to have actual customers, per se, since any group of paying users is still too small for Google to need to pay attention to.

① https://news.ycombinator.com/item?id=9912754

These are all little hobby projects for the advertising company.

Part of ecosystem. See how firebase today is integrated with analytics/ads.

Bullshit.

Any telco operator has millions of customers and it handles support just fine.

Any bank has millions of customers and it does support just fine.

Google enterprise products (Cloud, app engine, apps) are nowhere near that number. They have no excuse for not having support.

You must be living with different telcos and banks than the rest of us. They often frequent the lists of most hated companies, for example [1] has good representation from both industries.

[1] https://finance.yahoo.com/news/america-most-hated-companies-...

I don't have much to complain about all the providers I had in my life.

Sure, I had to wait 15 minutes on the phone a few times, or send letters now and then. Nothing out of the ordinary.

I think you misunderstand me. I’m not saying Google couldn’t have acceptable support. I’m saying they don’t have to; there is simply no perceptible downside to them if they don’t.

That is true for all their free public services. They are build without support. Can't support 1 billion free users.

That is untrue for all entreprisey services. They must have support.

What do you mean, “must”? They seem to be doing fine without it. Google will still have enterprise customers, no matter how bad their support is when it is really needed.

I think you've mischaracterised things a bit in your post - not sure if this is just because you're repeating what you read elsewhere? Anyhow, regarding the bullet points:

1. Google Wave was released an experimental consumer product, designed to combine the best of IM and email. According to the announcement, it was finally sunset due to lack of interest from the public. Personally, I was super excited about Wave - I even pestered an ex-Googler at the time to get me in on the preview. (I was in uni at the time). However, I had a lot of trouble getting other people to use it.

Did you personally use Wave yourself? Were you able to convert anybody else to using it?

Firstly - this is hardly apples to oranges. You're comparing a experimental consumer-oriented preview product, to an enterprise offering with SLAs. It would be like if we released a Star Wars game showcasing awesome new WebGL features - then decided to sunset it 12 months later.

Secondly - the project was open-sourced and then moved to under the Apache banner - an effort that in itself took significant engineering effort.

https://en.wikipedia.org/wiki/Apache_Wave

I'm not really sure what more you want?

In the case of Reader - this is somewhat personal, as I was a big lover of Google Reader as well. From memory, it was deprecated with around 12 months of notice, there were provisions for migrating/exporting all your data etc. It wasn't open-sourced like Wave, but there were reasons behind that (I can't comment). However, it was hardly the worst handled deprecation - and to be honest, the writing was on the wall for RSS for a long time before that.

2. You didn't provide any specifics, so I'm not sure what event you're referring to.

AFAIK, the only major price change was when Google AppEngine left beta/preview, and became an actual commercial product, and Google added SLAs, and other things etc:

https://groups.google.com/d/msg/google-appengine/Hluog1_a3n4...

The changes were announced at I/O that year (May), I believe, and the actual changes took effect in September.

Is that what you're referring to? Because even taken aside my Google hat, that seems like a fairly reasonable thing to do.

3. All of Google's enterprise offerings have support. I know, because I actually work in such a support team. For GSuite, support comes free as part of the product - you can contact us via email/chat or over the phone.

For GCP, there are commercial support packages available (https://cloud.google.com/support/) (similar to the AWS model).

> They've done it again and again. I'd say its the default way Google operates.

This is a pretty bold claim - can you cite any examples of this?

If this is something that personally affected you - please feel free to reach out to me. I can't promise I can fix everything, but I can definitely try to find out more for you, or see what I can do internally.

Disclaimer: I work for Google, but views expressed above are my own.

> This is a pretty bold claim - can you cite any examples of this?

I don't know if you have read the original thread, but there are a few examples of this: https://news.ycombinator.com/item?id=14356409

Also this: https://stackoverflow.com/questions/38959321/firebase-databa...

The parent comment posted in that thread, so I assume he has read it. While I read parts of the comment thread earlier when if first came up, I searched some key phrases and couldn't really find specifics other than the SO link you already mentioned. There was a lot of discontent and general ramblings about Google support being terrible, but I saw more people praising the paid support options and support in general for paid products than giving concrete examples of being burned.

I uploaded a small chrome extension a few months ago. Shortly after my developer account was closed for "violations of terms". I had just opened the account, and paid a nominal $5 fee for it.

I tried appealing a few times and was still not told what I did wrong or any way to get back on.

I got the impression that it was being handled automatically and that nobody could be bothered to write anything.

It's my understanding that Google has ramped up enterprise cloud support by outsourcing it. We ran into evidence of this recently. It's not helpful when the heavily accented, scripted Indian support guy at the other end is less technically informed than yourself, and there's no magic phrase to get bumped up to the next support level.

We're on Google Cloud Platform because of Kubernetes, and I'm dreading the day where we will need urgent support for something. My only consolation is that the Kubernetes community is very active, with lots of googlers on Github, Slack, Google Groups, and so on. We're trying to avoid using services outside of GKE in order to keep our vendor lock-in minimal (Kubernetes potentially allows us to switch vendors), and to stay within the realm of Kubernetes.

You have to face it -- Google's reputation has been severely damaged over the years, not just things like Reader, but also AppEngine, Google Apps support, arbitrary account shutdowns, and so on. Even innocuous, well-intentioned deprecations are hurting your rep these days, and you should consider avoiding this just to let your karma recharge; people will eventually forget and forgive, but if a new breaking change happens every month, they won't. (Contrast the situation to AWS, which has hardly deprecated anything since they started up, and manages to keep unpopular products running years after they became effectively obsolete.)

Commenting on mobile - but no, GCP support is not outsourced. I deal with and am good friends with some of those on the frontline for GCP support - that team works a few cubicles down from me, and from what I've seen incredibly passionate about providing stellar support.

Are you sure you didn't perhaps just talk to a Googler who happened to be Indian? It's possible I'm misreading tone on the go - but 1. Google is very multicultural, 2. As a non-white, I admit I can be a little touchy when people assume things about technical chops based on your race, or what accent your English is.

Regarding the fact you felt the person you dealt with wasn't well informed, that's not good to hear. Do you want to 1:1 details to me?

Also, I assume you've purchased GCP support right? That's a paid offering, and having dealt with them as an outsider, they're pretty decent (I mean, you are paying them...). I only ask because sometimes people post in public forums and assume they're dealing with official support. Our actual enterprise support teams are separate.

How about glass? That was hailed as the second coming according to the IO presentation, then they quietly dropped it.

>Did you personally use Wave yourself? Were you able to convert anybody else to using it?

I had Wave access (Gadgets were my primary interest in the platform) - With no disrespect to the team working on it, Wave was a muddled, slightly schizophrenic app that solved problems that didn't exist by trying to merge a whole bunch of stuff that already did.

Getting other people to use it was hampered by the problem of succinctly explaining to them what it did, and the (for a consumer product, at least) unintuitive UI.

That said, failed apps are the most interesting apps. As a dev it had a huge influence on my approach to problems and the merits doing one thing really well.

We used it on my team at work and it was fantastic for design discussions and the like. Incidentally, while slack, dropbox paper etc all target this space somewhat successfully, I think there's still a lot that could be done here.

Getting a huge user base for wave would have been easy - simply incorporate its features gradually into gmail, and make it so that if you're in a threaded conversation with someone with a wave server you have all the extra features, otherwise it's just normal email.

This is in fact what I assumed google were up to, since they were huge on the email front and they described wave as the future of email.

> From memory, it was deprecated with around 12 months of notice, there were provisions for migrating/exporting all your data etc.... - and to be honest, the writing was on the wall for RSS for a long time before that.

You couldn't export all your data easily unless you were only interested in starred item and subscriptions. If you cared about tagged items, you had to write your own code to fetch that or find someone else that did this (I know because I wrote such script).

It is true that you could access all of it and that you had about 3 months time to do so (not 12).

As far as RSS goes, I am still using it daily and have no problem finding sources except few places that really go for walled-garden like Google's G+.

> Did you personally use Wave yourself?

Thanks for the history lesson grasshopper. I have used wave - I was on the google Wave team here in Sydney. I was working on it the day the project was cancelled. (An intern then a contractor though, not a FTE, for reasons that are obvious in retrospect.)

I'm quite aware it was opensourced - I was part of the 6 person skeleton crew that opensourced it. Look me up in the codebase if you like - grep for comments in the Apache Wave codebase by @gentle. I think I still have commit access.

To catch you up, the opensource effort floundered. Unsurprisingly the 350k lines of dense, complicated, glitchy GWT java code we threw over the wall hasn't found the community of experienced developers it needs to fix bugs and add features. Its barely had a commit land in the last 3 years.

Like reader for you, Wave's cancellation was somewhat personal for me. Here's a secondary tragedy you might not know if you've only read Wave's wikipedia page: About 6 months before Wave was cancelled, Google bought AppJet (makers of Etherpad). Their team came to Sydney to work on Wave. After Wave was cancelled, for guessable reasons the AppJet team would stay at google for the next few years. They got stuck working on non-etherpad, non-wave projects. Their team eventually split apart and would never be reunited. The tragedy of google Wave resulted in not 1 but 2 innovative collaborative editors dying.

Do I trust Google to maintain projects I care about? No, duh. Picasa. Google Buzz. Protobuf spent years being unmaintained. Sparrow (the email client). Google Code Search. Orkut. GTalk's commitment to open messaging protocols. Glass.

If firebase doesn't get enough traction, do you think they'll keep it around, SLAs or no? I wouldn't bet money on it. I certainly wouldn't bet my business or professional reputation on it.

> All of Google's enterprise offerings have support. I know, because I actually work in such a support team.

Great. So why did google stop returning calls and emails from HomeAutomation? This thread exists because one of firebase's cofounders dropped the ball on an email. Why did a busy cofounder result in a complete communications blackout to a paying customer?

https://medium.com/@contact_16315/firebase-costs-increased-b...

There's an old piece of wisdom that the only way to get google to fix something is to either know a google engineer or complain loudly enough on social media. Yesterday that saying seemed to still be true.

I'm sure you work really hard to fight this problem at Google. Thankyou for that. I can't imagine its an easy job. But Google has a long way to go before I'm willing to lock my product in to one of their newish services.

That being said this was a great response and I appreciate it

Do you really think "Our sub-par standards got outed in a forum dedicated to our core target market so we did a spot fix and made some noise about how we will improve" qualifies as a great response?

A great response would be "We are aware our support is not inline with what a service like this requires. We will implement the following measures over period X"

This is more like "oops, sorry we got found out"

"Cloud" support in general is dreadful, especially given the extremely high premium you pay for *aaS services.

> SIDE NOTE: I don't get how a lot of people today don't understand that using vague language in apologies doesn't help with apologies.

It doesn't, you're right. But specific language in apologies doesn't help the company much during litigation, especially civil litigation when assessing damages. I realize that's not real sexy, but that might give you some perspective on your question. Imagine everything you write is being entered into a court transcript, and evaluated by a judge's clerk working on a recommendation for assigning damages. It is better to focus on delivering excellent service so things more rarely get to that point, but that's how apologies and announcements tend to get vague to the point of senselessness.

Meh. This is just a crisis-response PR move: "look at all the flak we're getting at HN, do something NOW!!! (And then go back to the support-slumber as usual)" Do you believe anything would change if it were not for the above post making it to HN frontpage?

Hi! One related question, and one offtopic question.

Do you happen to have any idea what actually happened internally with this? I ask this coming from the standpoint of "ouch, another example of ignored paying customers". Obviously this is a difficult question to answer generally, but extra detail about what happened has the potential to instantly pull this specific instance out of the generic "Google support is insufficiently human" bucket, which might be interesting. (Please note that I'm asking this to get the other side of the story about this, I'm not trying to shoot the messenger :) )

OK, now for my offtopic question. I think you're probably the perfect person to ask this.

https://github.com/HackerNews/API (linked from the bottom of every HN page except the add-comment page) describes HN's Firebase-based API. The current API design tends to require a lot of discrete requests to get at high-level information due to the fact that it doesn't support batching (and the page acknowledges this, with "It's not the ideal public API, but it's the one we could release in the time we had.").

Now... that page also says "There is currently no rate limit."

For some time I've wanted to track page votes over time. These are not logged, so this operation is necessarily very realtime. There are lots of posts, and when one of them goes viral the vote goes up very quickly. Perhaps you can see where this is going :)

If I wanted to try and overcome the poor API design by requesting individual items every 500ms or 250ms, or 100ms.... or 50ms......

a) at what point am I likely to get hard IP-blocked? (I'm also wondering how bad it/I would be if I used a bunch of different IPs, at least in terms of technical load.)

b) what rate should I tend to prefer so I can be nice to HN (I'm not sure what tier they're on)?

Thanks for responding to the issues and admitting the support has got worse since your association with Google. That's a big admission to make (are you sure Google doesn't punish people for revealing such things?). You might also want to tell 'other teams' in Google what's the perceived image of their support outside. What Google Support means for me for most of their services is - some 'internet forum' on which a few unpaid workers are trying to help others who are equally clueless, in return of some internet brownie points.

Thanks for the comment

Until this[1] is fixed I'm not sure that Firebase would meet my needs for many projects. It's such a fundamental requirement to be able to query multiple fields without creating a mess of permutations of each field.

[1] - https://youtu.be/sKFLI5FOOHs?t=541

In particular I would love to see the query size change in the real-time database. This could either be achieved by compressing the JSON response, or by calculating the traffic differently.

As a user of Firebase about a year ago I saw much, much higher egress traffic from the Real-time database than I expected. To be specific, for testing purposes I set up a note-taking app, which (being a test) had the database size of 31kb. But here comes the weird part: by querying the database exactly 10 times, my console was showing 24.5mb of traffic, which is much higher than 31*10=310kb/0.31mb.

Ultimately this caused me to change tracks to an open-source alternative, but if changes would be made on that front I'd be willing to take another dive into Firebase.

It looks like you need an offline-first solution like pouchdb or rxdb. The traffic would have been around 31kb, no mather how much you query.

I really appreciate the input, however the note taking example mentioned above was just a test project of mine. The actual project needs real-time synchronization, which isn't something I was expecting pouchdb/couchdb to be capable of.

Hi, Tom from the Firebase Realtime database team here. To address the difficulties of figuring out where performance issues were, we made a database profiler (https://firebase.google.com/docs/database/web/profile). I expect that would help pinpoint where the issue was. My informed guess would be an index was not setup as you expected and the querying was being done client side, or my next guess would be the client was rapidly connecting and disconnecting. (check clientside with Firebase.database.enableLogging(true)).

The profiler excludes SSL overhead, but you can see usage inclusive of SSL in the webconsole.

Reply on news.ycombinator.com