WatermelonDB, a database for React and React Native apps | Hacker News Reader

WatermelonDB, a database for React and React Native apps

github.com

118 pointsjinqueeny51 comments

Why not use SQLite directly instead of this extra layer?

Looks like it's mostly to provide a lazy loading wrapper for SQLite to React

Because it's very cumbersome, interfacing between Javascript and SQLite, and mapping between database records and Javascript objects. And it doesn't easily fit with the React way of doing things without some library to abstract away all the wiring.

I started building a hybrid mobile app (not React) with a large SQLite database about 4 years ago and have kept working on it ever since, and I've found the data storage aspect of the app very painful to maintain and extend.

I'm now starting to rebuild it using React Native, and a storage plugin like this could be perfect.

I'd been looking at Realm, but haven't gotten as far as committing to it yet, so I'll take a close look at WatermelonDB too.

I'm not buying the "too cumbersome to use SQLite" argument.

This still seems like a large unnecessary abtraction.

Sincere question: do you much work in React/React Native with large SQLite databases or other large data stores?

As the sibling (and better) comment[1] to my original comment points out, this library makes the data automatically observable throughout the React app, making it much simpler to update views in response to database changes.

I can understand you might not understand the value of this if you hadn't done a lot of work with React apps that have large data sets and where data changes need to be reflected in multiple different views.

[1] https://news.ycombinator.com/item?id=17958822

Why don't you try linking react with a large Sqlite, and later build a wrapper of your own, and then release it in HN next year as, pinappleDB?

SQLite gives you just the "data fetching" part. But if you're building a React/RN app, you probably also want everything to be automatically observable.

For example, if you have a todo app, and you mark a task as done, you'd want the task component to re-render, the list to re-render (put the task at the bottom), and update the right counters (the counter of all tasks, and a counter of all tasks in ta project), etc. I hope you're getting the point.

Watermelon is an observable abstraction on top of SQLite (or an arbitrary database), so that you can connect data to components and have them update automatically

So is it fair to say its a React specific ORM, that automatically makes the mapped objects observable (an OORM, if you will)? Just trying to understand the level of abstraction that you're going for.

That's a fair description, yeah!

Does that mean that if user A makes an update to the database, then user B (on a different computer) will see the update?

No, Watermelon is a local app database. But you can plug it into a sync engine to synchronize with the server (and then from the server to another device) — it's up to you

Thanks for this concise example! This helped me to understand why I should try it out, and led me to demoing this myself. I'll report back my findings

Because then you don’t get to reinvent the wheel.

It's easy to be cynical on HN, but to the best of my knowledge, no other solution for React Native checks all the boxes we need.

And it's not like reinventing the wheel (it still uses SQLite), more like inventing better suspenders for the whole undercarriage ;)

Isn't 10,000's of records a bit to low for a database ?

A mobile app database. It can probably scale to hundreds of thousands too, but very few apps and very few users of such apps would actually have THAT much data.

The point is that most current solutions for React Native apps start being super slow when you get to 1000s, or have other issues (like using SQLite directly doesn't give you observability)

This looks really good, actually. Having used SQLite, I waited a long time for a good, functional, easily-debuggable DB for React Native.

Realm didn't cut it on the debugging part.

I'll be really curious to hear your feedback on GH if you try it. We've been using it internally in dev for a year, but we've just released this as open source, so there's probably a ton to improve on.

When SQLite is not available in the browser, can it fallback to using use an Adapter which just implements a (low performance) DB in memory or storage?

edit: link to Adapter docs https://github.com/Nozbe/WatermelonDB/blob/master/docs/Imple...

Yep, if you can support some (relatively basic) queries, and CRUD operations, you can plug in essentially any database, and bridge on any platform. Think Electron, SQLite on native macOS, heck, you could use Realm as the underlying database if you so preferred...

How does WatermelonDB compare to Realm?

I'm not an expert at Realm, so I might get some details wrong, but briefly:

- You can't run Realm on the web (not as a database, offline), but you can with Watermelon

- Realm is _a_ database, Watermelon is a more universal database framework that can be backed by any underlying database via an adapter

- Realm is more than just a database, it's also a whole app development platform. And it's designed to sync with a proprietary Realm cloud service. Watermelon is just a local database, but it tracks changes in the database so you can plug it in and synchronize with an arbitrary backend

- I heard that Realm (on React Native at least) doesn't perform super well when you get to many thousands/tens of thousands of records, but i haven't tested it yet myself, so take it with a grain of salt. Watermelon always does lazy loading so it doesn't really affect performance how many records total there are.

Would this work as a kind of 'rxjs-persist', similar to 'redux-persist'?

Looks like an awesome abstraction layer!

I'm curious how this would handle syncing like Gun (https://gun.eco/), and an example of how it might work with Vue.

Not familiar with either Gun or Vue. I'll look into the former, and as for the latter — Watermelon does not actually depend on React. If one can supply a way to hook an RxJS observable into Vue (a la higher order components in React), it should work perfectly fine. Pull requests welcome!

Looks like it's just an ORM for SQLite. Isn't SQLite too large to be included in the web?

Yeah :/ There used to be this thing called Web SQL (which really… was SQLite)… but… people didn't like it and it didn't catch on. So on the web, Watermelon uses LokiJS, which isn't perfect, but still works well.

> but… people didn't like it and it didn't catch on.

Mozilla didn't like, developers liked it. Mozilla's excuse was "which SQL spec?" which could have been solved with a bit of work. Same as the File System API proposals by google. I'm sorry but indexed-db is not a replacement for these 2. indexed-db is slow and inefficient when it comes to data storage and is horrible to query.

Agreed! IDB is simpler in some ways, but having a real relational database is a lot better for many advanced applications — and also can be more easily used for porting to different platforms, since you can run SQLite on any native platform

IndexedDB used to be slow, but is it still yet? Not so sure, performance does not seem that bad (see http://reyesr.github.io/html5-storage-benchmark/ ) Do you have some benchmark that shows it is indeed slow as of today?

Wanting a specified query language for a database API isn't really an excuse IMHO. (and if it really were just "a bit of work", one has to ask why the proposers then didn't bother to just do that. As cool as sqlite is, tying browsers to a specific version of it seems questionable)

WebSQL was killed by Mozilla and Microsoft. A huge idiotic mistake, probably related to the anti RDBMS NOSQL sentiment and fashion at that time. Browsers should ignore this crazy decision and keep (or add) SQLite in the browser - is still in Chrome but deprecated. SQLite is faster and way more capable than Indexdb and free and open source and runs on all platforms and devices - at least keep it as an option.

Mozilla and Microsoft both agreed that the web needs standards that are easy to implement multiple times, otherwise you wind up with another IE6 compatibility nightmare situation. Even though SQLite is open source, there's still only one SQLite. If browsers had to embed SQLite, then the web would have to deal with compatibility issues from SQLite bugs for decades.

It doesn't matter that IndexedDB is less capable than SQLite, it matters that IndexedDB is a well defined standard that can be easily implemented and optimized independently by each browser. It was polyfillable on day one on Chrome's SQLite install, and the other browsers were free to choose where and how to implement IndexedDB.

Also, IndexedDB has gotten quite fast in recent browser versions and is only getting faster, thanks to the browsers being able to compete on how they optimize it (versus all of them trying to pass competing PRs to SQLite, or working on increasingly divergent forks of it).

A goal for IndexedDB was that it might be low level enough that you could build RDBMS database APIs on top of it (whereas you're hard pressed to build certain types of key/value stores or document stores more directly on top of SQLite). The Browsers weren't anti-RDBMS, they were anti-lock-in to a single vendor's bugs. Microsoft learned from their IE mistakes, and don't want Chrome to repeat them (as hard as Google sometimes seems to be trying to make the same mistakes).

Yes standards are great and SQL is a fine example of this, but the standards don’t make anything - developers make stuff from tools and the software they write or build upon. It is stupid to handicap a browser, to throw away an incredibly useful, already made, battle tested piece of software (SQLite) just for the sake if the almighty standard! And to replace it with a feeble slow key value db. So we now have the ridiculous situation of umpteen man hours being wasted reinventing the wheel, trying to build relational databases on top of IDB, it’s crazy.

SQL is a terrible example of a standard. Half the major databases that support the "standard" can't even agree with each other on what the syntax is to window the number of returned results.

SQLite is partly so battle tested simply because once you start down the path of relying on it, it becomes tough to shift to another SQL "standard" database, because of tiny differences in dialect, and sometimes huge differences in performance characteristics.

MySQL/MariaDB and PostgreSQL are also, open source, battle tested pieces of software. Why would a browser stop at SQLite and not allow developers the power of an equally tuned, "full" SQL database, with additional power/capabilities/storage management capabilities? Would a browser have to reimplement SQLite's dialect on top of it to support that as an option? Would a browser need to implement SQLite bugs on top of PostgreSQL? Would you want to write a website that does feature checking to see if WebSQL is backed by SQLite or MariaDB on the user's machine and adjust its SQL queries for the different query engines?

Meanwhile, you can look at Redis, Cassandra, Riak, BigTable, etc: we have the technology to build really scalable key/value stores that can scale and perform incredibly well. We also have the knowledge of building higher level APIs on top of that and getting those to scale and perform well. IndexedDB's slow period was a brief adjustment period, and benchmarks indicate that calling it "slow" or "feeble" today isn't accurate. We're certainly on the path where IndexedDB's performance is only steadily improving, and again, we have the technology to insure that it can scale and perform extremely well given the investment.

IndexedDB is a more than adequate compromise to the tar pit that is decades of weirdly incompatible SQL dialects and database engines, and to the security nightmare implications that if there was a 0-day bug found in SQLite code you could take down every browser on the web with it had SQLite been blessed to be such a "standard" that every browser just integrated it directly.

On the security question of using only a single dependency for all browsers, an infinite loop exploit in SQLite appears: https://news.ycombinator.com/item?id=17964243

There's a very promising attempt to get WebSQL-like features to browsers: Relational Database Proposal. Proposal tries to solve limitations Lovefield library authors had to deal with. (Lovefield powers GMail Inbox if I'm not mistaken.)

https://github.com/arthurhsu/rdb

https://github.com/google/lovefield

I worked on an application which was essentially a multi master, offline capable, synchronizing app that utilized websql heavily. Websql wasn't perfect (it was essentially an old version of sqlite, I think), but as a whole it worked pretty well. Was sad to see it die.

In WatermelonDB's case, I bet most people will run it using React Native (ie mobile apps for iOS and Android), so that wouldn't really be an issue.

On the other hand, sql.js (https://github.com/kripken/sql.js/) packs to about 2,6MB (see http://kripken.github.io/sql.js/js/worker.sql.js). I don't know about performance under heavy use of that. I don't know what SQLite backend WatermelonDB uses from ReactJS either.

Great work, reminds me of minimongo. Any plans to make it easier to sync with a server db?

Yes! Planning to release a few helper functions and document exactly what the procedure is to synchronize with a server. Internally, we know it works.

Would it be possible to write an adapter to use this with postgresql on the web?

It's possible, but Watermelon is specifically designed to be a local database, not meant to run on a server.

can I use this in conjunction with graphQL and middleware such as apollo client? so essentially, use the middleware to query my DB and use watermelon for local storage?

not very familiar with apollo, but I don't see why not. But just be aware that Watermelon works best as a full local database copy (that synchronizes with the server), not just a cache.

The killer feature, quite understandably absent, is sync.

See: https://news.ycombinator.com/item?id=17960347

Sync is mostly there, but undocumented :/ Coming soon…

In academia we say "not published, not done". Perhaps the open-source community could benefit from "not documented, not done".

You’re not wrong

But am I right?

Reply on news.ycombinator.com