Hacker News Reader

Comment by paulsutter | Hacker News Reader

To understand what happened to RethinkDB, you want to contrast it with MongoDB.

RethinkDB is amazing software and a real breakthrough. And the audience of deep thinking people who can really leverage the event model is ... limited. Today.

MongoDB is no breakthrough, but there are legions of people who just want to plug something together with PHP or Node. Given what they're doing, it works fine.

I sincerely hope Rethink can turn into a vibrant open source project and I'm willing to donate money to that. Given sufficient time, the world will learn to use it. And until then, it will grant superpowers to people willing to invest a little effort to understand and leverage it.

View on HN

FWIW I found Rethink easier than Mongo, especially the query language. ReQL is actual JavaScript—Mongo's query language always seemed weird to me.

Mongo was "there" first and generated a lot of hype through sales and marketing. For a while and probably still now, a lot of Hackathons at college campuses are sponsored by Mongo and you can usually get a prize for using Mongo in your project.

Rethink never marketed themselves too much—it was one of those things you found out about in HN or from another dev.

I'm clueless about business stuff so I don't mean to say Rethink should've marketed heavily like Mongo did, but that's definitely a factor. I also think Rethink should've done Horizon earlier and offered paid hosting/deployment for it and Rethink users.

I admit I don't have a lot of experience with MongoDB, but RethinkDB was really easy to plug into Node. While we didn't support PHP officially, the PHP RethinkDB driver was maintained by RethinkDB's engineering manager, Daniel Mewes, and it looked pretty solid.

When I started at RethinkDB, it was before changefeeds were a thing. I always thought the "killer feature" was, well, ReQL. It had the qualities of an ORM/DB Builder in your native language of choice. There was a poster on the wall that said "Databases should be indistinguishable from magic"; I always thought that was a perfect pitch to the RethinkDB audience. I understand why they canted toward pitching the realtime aspects first and foremost after changefeeds came about, but I thought it had great power even if you weren't focused on realtime/event-driven aspects, and perhaps that should have been emphasized more.

In my opinion rethinkdb is super easy to get going and use. The low touch maintenance is what drew me originally.

However, learning to live without transactions takes some rethinking of the approaches taken. It's totally doable but it may not come easy if you're used to transactions in an RDBMS.

Of course in today's distributed multi service multi database world, learning to live without transactions is becoming more and more important anyway.

if "Databases should be indistinguishable from magic" then somehow you should be able to get transactions

You can get transactions, but at a cost. Distributed transactions exist, they're just slow (and network-chatty). RethinkDB decided not to go that direction, I guess.

Disclaimer: I have a bro crush on Ben Horowitz.

Exactly, they gave it a good go as a company but their burn rate was likely too high/too soon and differentiation insufficient (whether lack of actual novel features and/or communicating advantages in marketing). For example, a polar opposite of something like Couchbase which is used by giant enterprise shops to move away from Memcache and/or CouchDB.

It was also competing in a way Peter Thiel and Paul Graham warn about. Reinventing something without a vital quantum of improvement is like trying to juggle while climbing Half Dome.

I hope Rethink lives on as open source and adds MongoDB support. If they can hustle to encourage more people to try it by selling its obvious advantages, they have a slim but nonzero chance of rebooting.

Next time, gotta find more paying, alpha customers first and then start building. Then, try VC only after being capital-constrained when hiring/expansion/development is a blocker to realizing imminent profit. New category brands and segment-optimized products are riskier but more likely to be able to set up moats with defensible, competitive value.

You nailed it with this "try VC only after being capital-constrained when hiring/expansion/development is a blocker to realizing imminent profit".

Don't raise money for an idea, raise money for a business that is capital constrained. VC is only one kind of funding you can try to grow with.

The best source of funding is revenue.

For those not "in the know," what capability is that? Everything I hear implies a focus on ease of use: Easy replication/distribution, JS API, single number type (float64), usefulness when teaching novices, etc.

RethinkDB's niche is allowing a service to listen for changes to a dataset. As in, your Node application could listen for published events by Rethink, rather than your Node application long-polling the dataset for changes. "Real Time" data analytics can then be performed.

Its other niche is easy to deploy replication and sharding that actually works, actually scales, and does not lose data.

Pretty much anybody can appreciate that, right? Not just the "deep thinking people"?

I've never seen that in any db I've worked with. If that's true, that's pretty huge.

It is true. It was NOT true a year ago, but it is now. I pushed it hard, Aphyr pushed it hard, and the RethinkDB team worked very hard addressing every issue we found. That's why RethinkDB failing is so tragic.

Yeah it feels like they died right after getting everything right.

Damn, I'm simultaneously excited and sad now. I hope it survives as open source so I can use it for my next project.

Firebase works like that. The polling of databases is a really obvious limitation of existing databases once you see the alternative.

(I work at Firebase)

Has Firebase been through Aphyr's Jepsen tests?

Seeing as how Aphyr can't deploy his own instances of Firebase to simulate network partitions, I don't think it has.

I searched and searched, but I couldn't find a download link on Firebase.com. Help?

I think Firebase is a closed source program that can only be used via "the cloud", i.e., via a web application. It's not something you can run yourself. In contrast, RethinkDB+Horizon would have soon provided similar functionality to Firebase, but with the option of self hosting.

Well it's easy to shard when you don't support joins or foreign key constraints.

Btw, as a person who works on Firebase, perhaps you can answer this question that has been bothering me about it. There's no way to model a foreign key relationship that's traversable in both ways without trusting the client application to update both directions, right? Isn't that a bit of a dealbreaker for anything other than a toy app? I would assume if you wanted to make something that's actually reliable, you'd have to run all your mutations through your own personal server instead of letting clients mutate Firebase directly, right?

[edit: this is an irrelevant remark] RethinkDB supports joins: https://www.rethinkdb.com/docs/table-joins/ "Like many traditional database systems, RethinkDB supports JOIN commands to combine data from multiple tables. In RethinkDB joins are automatically distributed—a join command is automatically sent to the appropriate nodes across the cluster, the relevant data is combined, and the final result is presented to the user."

I'm talking about Firebase :P.

Rethink joins aren't consistent (in fact, no Rethink multikey transaction guarantees any degree of consistency), so even though the original conversation was about Firebase IMO it is just as applicable to Rethink.

You can use it as the mobile device connector as clientside auth, push, latency compensation, presense and offline persistence are big features on their own. For sensitive joins, have a server side joiner with administrative privileges. Still, it's work that should be easier I agree. Firebase's strength is not what other DBs can do, but what they cannot.

I'm curious what Firebase users do in real life. Do people realize this is a problem?

I can imagine making an app like that where all mutations go through my server and reads go through Firebase. How would this interact with the offline persistence feature though? I suppose you could allow the client to do some simple mutations like editing text fields. For the mutations that must go through the server, though, perhaps you could do optimistic local updates?

The latency compensation through the sdk is optimistic writes. If you shortcut to server, you have to reimplement that so that's not a good solution for many cases. It would confuse the offline persistence too.

Instead, your server should listen to the Firebase, rather than have writes go a separate path (which loses lots of useful features). I think the term now is a client-database-server architecture.

it's not actually a problem for 90% of apps. It's usually in the users best interest not to break their relationships in the db. Think chat. Think info services like Pokemon. Cross table transactions are relative minority in the space of useful apps that's can make money. For those more complex domains, you need technical know how, and Firebase is a useful system component in a larger architecture.

As soon as you want to view something through a different direction you run into this problem. Sure, you can embed chat messages into a conversation, but what if you want to look up every message by a particular user?

I would be surprised to find that 90% of apps don't have this kind of requirement. Pokemon is easy because that data is static.

Viewing all posts by a user is solved with classic denormalization. You maintain a separate index of post by user and do two writes per chat msg. It's a case where a user has no incentive not do the second write so it be fine to implement clientside. If however it was business critical, you would have a server side process listen to all chat messages and write the user to message index with the admin account. That index would essentially be eventually consistent.

Concur. We've been running a small cluster for about a year and we've had the same experience thus far.

I've never done it, but I think it's possible to do this in Postgres as well using LISTEN and NOTIFY. You may want to do all of your database mutations through functions that ultimately call NOTIFY for this to work though. But uh, that's how you'd do it in postgraphql or postgrest anyway.

https://github.com/calebmer/postgraphql

http://postgrest.com/

You can definitely use LISTEN and NOTIFY to watch for changes in Postgres. Here's a presentation I did on that last year, though oriented more towards web apps than analytics. https://www.youtube.com/watch?v=PsorlkAF83s

For analytics I would want stronger guarantees than LISTEN and NOTIFY alone can provide. I explore options for that here: http://btubbs.com/streaming-updates-from-postgres.html

isnt that what Realm is trying to do on the mobile side ?

"Worse is Better" triumphs again...

Just going out on a limb here, and I know neither MongoDB or RethinkDB (nor the companies behind them) very well.

But, from what you're saying it almost sounds like that RethinkDB would be a great companion to MongoDB. And if so, that the company behind MongoDB might be a great custodian for RethinkDB.

That thought fills me with dread. Rethink is well engineered and has a great developer experience. Mongo isn't and doesn't.

All these comments are starting to make me feel stupid. I picked mongo for a side project, and it's been really nice so far. I have a hard time imagining how one would use it for a complicated and/or large schema though. I assume that is just my newness to it.

Nope, you have it right. Mongo is a lot of fun to play with out of the box. Once you have nontrivial data models, redundancy and load, not so much.

Don't feel bad for not understanding how to make it work for a big project, it just doesn't. You can try make it work for a big project, but at that point you are just implementing database level code in your application.

Better to just use something like PostgreSQL. Contrary to popular belief PostgreSQL is actually a better document store than MongoDB via it's jsonb type, operators and GIST/GIN indexing.

I'm not entirely to sure if I understand your comment, however, what I mean is;

The company behind MongoDB is a database company so they probably have some sort of affinity and appreciation for a product like RethinkDB, and, they are commercially successful (or at least, it appears to be that way).

Just because, hypothetically, MongoDB could be a custodian for RethinkDB don't mean that all of a sudden RethinkDB becomes MongoDB.

As Paul pointed out, both solutions seem to be two different products covering two different use cases and audiences and from what I can imagine, they might actually complement each other.

MongoDB was much more breakthrough than this little thing called rethinkdb. Hell, people even create the term NoSQL because of it. Maybe the failure of this project came from this dellusional thinking I keep reading in all posts about their failure.

MongoDB coined the term maybe, along with webscale and a whole bunch of meme.

But "NoSQL" atleast as it'd defined has been around forever, it predates SQL because everything before then was trying to work out how to store bits on disk as efficiently as possible before relational theory really found it's way into storage systems and revolutionized things by focussing on modelling.

Even putting aside that fact it was a terrible database then, it's a slightly less terrible database now and everything it can do is done better by other databases.

Reply on news.ycombinator.com