Hacker News Reader

That's not the only limit btw. If any node (say "/users/") crosses 'some' number of subnodes ("/users/a", "users/b") then you cannot do any queries on the node itself. Like I cannot get even get the IDs of the subnodes ("a","b",...).

I also got similar advice to shard my users or something.

So right now, we have crossed that limit and are unable to know how many users are on our system. Their server just fails and takes down the DB for 10 or so mins if I do that query.

Firebase is good for MVPs and prototypes but not at all scalable.

Wow.

Running a query takes down the db?

That sounds like a major problem. How can that happen?

Maybe returning incorrect or incomplete results due to sharding... But taking the db down? That's very... Unexpected.

Nope. A clear server crash with "Internal Server Error" and the DB being totally unavailable for 10-15 mins. Apparently it's 'normal'.

Never used firebase but for what it's worth you can take down most databases with a bad enough query.

You are right. But we usually have multiple ways to query our data in a DB. Not many ways to do that with the limited Firebase API.

In the RDBMS world, you can take down pretty much any database by giving someone in accounting a copy of Crystal Reports :)

How, specifically? Something like a very complex query joining too many tables, or maybe a full Cartesian product of n > 2 tables?

> Their server just fails and takes down the DB for 10 or so mins if I do that query.

I don't know if you are aware, but their DB can't handle more than 1000 requests/sec, so if you are iterating through a list of nodes and requesting data for each one you can hit that limit (not a good practice, but sometimes you have to). Additionally, once you hit that limit, the DB slows down but keeps accepting requests meaning, if you keep hitting it, even at a slower rate you make the backlog worse. Seriously, be very very careful not to go over that limit, we found out the hard way.

Iterating would require me to first get atleast a 'shallow' list of keys for that node. But even that one REST query for shallow list of all keys crashes their server instance.

I am not even sure how I should get keys for all my user nodes anymore.

I tried doing it from an offline JSON backup they generate. But that one giant 100GB+ JSON is impossible to parse with any available tools.

Hmm. Congratulations, you've nerd-sniped me this morning!

This sounds like a classic use case for a streaming parser. The data is a mile wide and an inch deep, so at any point the memory requirements should not be too high.

What do you want to do when you've parsed it? Insert it into a real database? Iterate over it? Would simply turning it into a list of user IDs one per line suffice?

Hello. The JSON dump looks something like this:

  {
    users: {
      "userid1": {...},
      ...
    },

    ...
  }

I have tried jq stream parser to split the big dump into files like: - users.json - chatrooms.json - ... So I can then work on individual nodes.

But jq fails silently after 12-24 hours of processing. I am still researching this in free time.

If I can just get the keys (like "userid1") I can do the rest from firebase itself.

The parser would have to parse the whole thing before being able to split it.

You could to the splitting yourself (it's just plain text) and create multiple files whose contents is just an array in the format:

```

[

{},

...

]

```

Then you can use JSONStream to load each of those files individually and map/reduce on the contents.

I would think something like this would work:

  cat input.json | jq -c --stream '. as $in | select(length == 2 and $in[0][0] == "users") | {}|setpath($in[0][1:]; $in[1])' > users.jsonlines

Output would me a file that looks like this:

  {"userid1":{"name":"user1"}}
  {"userid2":{"name":"user2"}}

I think I have tried something similar. I will try this one too. Thanks a lot!

You could loop through the file counting braces, storing line numbers. Then split the file along those line numbers. The smaller files might not have the exact formatting you need to run through a parser, but you should be able to manually adjust it then, hopefully.

It's not impossible. You can do it pretty simply using https://pypi.python.org/pypi/ijson

> The problem I have is that someone took this cool project and basically lied to the public about its capabilities.

Oh come on, it was a startup. The same people who made the tool told the lies (if that's what they were). Now under Google, this is still the case - the original founders now lead the Firebase department if I'm not mistaken. I'm pretty sure they have influence over the marketing messages.

> Oh come on, it was a startup.

And then people are surprised when I advise them to avoid startup products.

It's the classic tragedy of the commons; lying your way into an acquihire is a win for the successful founders, and a loss for the everyone else in the ecosystem, as more and more people grow sick of being played that way.

funny how you're right. i just got an api shutdown notice from stormpath (yet another product that supposed to save us developers): https://stormpath.com/

When was that announced?

You basically have 2 weeks to implement user management functionality? An acquisition so hot they basically say fuck you to all customers. Amazing.

The announcement post[0] seems to be from March, which would give one about 4.5 months to migrate.

--

[0] - https://stormpath.com/blog/stormpaths-new-path

You misinterpreted my comment on purpose. I didn't say "it was a startup, so it's ok" I said "it was a startup, so the same people who built the service did the marketing".

Also, Google's Firebase acquisition was many things, but not an acquihire.

Wow, didn't know that. :/ Guess it is really only useful for prototypes.

I'm curious, what DB tool will you now consider using instead?

Same here. I've started a prototype using firebase, however, I'm concerned about issues like this and the fact that sometime in the future I get a shutdown notice and I will have to migrate.

I can roll my own using Rails but I'm trying to avoid that.

good old mysql. it is the cockroach of software.

> at 100k concurrent users

That is a huge number of concurrent users though.

The risk of building on a platform that you know has that hard cap isn't worth it for many ventures though

Depends on your perspective, but it doesn't matter as it's not the unlimited they advertised.

Our app has multiple millions of users and we wanted to add a small feature which we were going to use Firebase to back. We ended up having to go back to the drawing board when we started a gradual roll out and hit that limit pretty quickly.

same thing bit me. also the onDisconnect events are not guaranteed to fire, so you get orphaned stale data everywhere, and need to run manual clean up scripts.

It is definitely true that pagination could be much easier. The way it currently works however seems fine to me: You query a node with say the rate limit of the first 30 children. Then you store the key of the last result, and use it as the starting point for the next limited query. This way you can easily query a node with 1000+ children without overloading the client.