Dflat: SQLite FlatBuffers | Hacker News Reader

Dflat: SQLite FlatBuffers

github.com

123 pointstosh17 comments

Very interesting concept, but I'd recommend people check out GRDB[1] if you're in search of a mature persistence layer for your iOS apps. It's a modern SQLite wrapper with a lot of conveniences for application development like value types, Codable mapping, Combine observables, etc.

I've deployed it in production on a few apps so far, and it has been a real joy to use.

1. https://github.com/groue/GRDB.swift

Sure. I like the idea of FlatBuffers. I think GRDB is also good as I'm reading more and more about it. Thanks for sharing.

> Row id

> We can uniquely identify an object, even if it is deleted later, without worrying a new object could occupy the same rowid

I don’t know anything about mobile and not sure if it applies here but if you call vacuum on sqlite, it rewrites row ids. Is there a way to prevent that? If you have a column “PRIMARY KEY INTEGER”, it is your row id and it won’t change on vacuum, otherwise if you rely on automatic row id, you may have a surprise later.

It's not just on vacuum. Rowids are reused whenever stuff is deleted from the end of the table:

    sqlite> CREATE TABLE t(x);
    sqlite> INSERT INTO t(x) VALUES ('foo'), ('bar');
    sqlite> SELECT rowid, x FROM t;
    1|foo
    2|bar
    sqlite> DELETE FROM t WHERE x='bar';
    sqlite> INSERT INTO t(x) VALUES ('baz');
    sqlite> SELECT rowid, x FROM t;
    1|foo
    2|baz

This is true even if rowid is aliased by an INTEGER PRIMARY KEY. Whenever a row is inserted, the new rowid is simply 1 greater than the largest rowid in use (or 1 for an empty table).

But Dflag is using:

   rowid INTEGER PRIMARY KEY AUTOINCREMENT

which prevents ids from being reused. So this isn't actually a problem for them, but the documentation could be clearer about it.

> This is true even if rowid is aliased by an INTEGER PRIMARY KEY.

Thanks. Just to be clear, I think this is the scenario when you don’t set your primary key but other columns in your insert statement.

CREATE TABLE t(x INTEGER PRIMARY KEY, y);

INSERT INTO(y) VALUES(‘value’);

Otherwise, if you set primary key in your insert statements, you’ll be fine I guess.

I’m not worried about reusing ids but if ids change e.g on vacuum, it is a disaster. Because people use row id(via last_insert_row_id()) assuming it won’t change but vacuum changes it and now you have a “dangling id”, if you fetch data again with that id, either you won’t find the row or get some other random row.

I may omit details in that paragraph, sorry! Autoincrement was used to prevent rowid reuse (and it is explicit primary key, where the properties you marked as primary in flatbuffers schema is just unique index), SQLite documentation claims that should be enough: https://sqlite.org/autoinc.html

Thank you, sounds great.

> Dflat at the moment requires Bazel.

Very few people use Bazel for Android or iOS. Anyone not from Google care to let us know if you're using Bazel for mobile development?

The irony with Bazel and Google, is that not even Google's own teams bother with Gradle for Android, the large majority seems to prefer use Bazel as per public projects.

Seeing as Bazel comes out Of Google, there's not a great deal of irony there.

There is, the Google own teams cannot be bothered to use what outsiders are forced to use when targeting Android.

I think it's more complex than that, bazel (and its predecessor blaze) were created to make monorepo development at google scale work. They use bazel/blaze not because it works better or more easily for Android, they use it because their entire engineering culture depends on an enormous monorepo of sharing code and rapid changes across the company codebase. There are some great papers they've written about bazel that go into these issues and more.

My company uses bazel for Android development. We tried to use it for iOS but ran into issues with cocoapods.

Pinterest uses bazel for their iOS app: https://medium.com/pinterest-engineering/developing-fast-rel...

Dflat's compiler requires Bazel to build because it is easier to reference to flatbuffers' source code with Bazel than with Swift Package Manager. The runtime itself (which you delivers with your app) can be built with either Bazel or Swift Package Manager (as suggested through CI). It potentially can also be built with CocoaPods, I simply don't provide an official package on CocoaPods at the moment.

This is an alternative to Core Data. The DFlat tool generates Swift code from a FlatBuffer schema and persists objects in SQLite during runtime. Core Data objects are defined in Xcode using the Interface Builder.

This looks cool but I don't do iOS native so can't give it a spin.

It's been a while since I've looked into flatbuffers (back when I was into 3D I wanted an efficient binary serialisation framework) - but I kind of like the idea of automatically creating a flatbuffers layer on top of a relational DB, something PostgREST like - but instead of going flatbuffer schema -> generate DB schema I'd go the other way -> generate flatbuffer schema from existing DB schema and allow some manual editing/mapping.

Are there non-mobile use cases for something like this?

Reply on news.ycombinator.com