SQLiteData: A fast, lightweight replacement for SwiftData using SQL and CloudKit | Hacker News Reader

SQLiteData: A fast, lightweight replacement for SwiftData using SQL and CloudKit

github.com

56 pointswahnfrieden45 comments

GRDB/GRDB-Query (https://github.com/groue/GRDB.swift) is another solid library worth looking at if you're looking for libraries in this space, though it doesn't have CloudKit support out of the box.

The more SwiftData alternatives the better -- as it has a lot of rough edges, and Apple hasn't invested much in it since it's initial launch.

Thank you (GRDB author here).

It is not mentioned in the README of the repository, but SQLiteData wraps GRDB to access the database and get notified of database changes (the meat and butter).

GRDB is by itself a solid "toolkit for SQLite databases, with a focus on application development", with both high levels APIs for everyday coding, and expert SQLite features for the demanding developers. Many apps rely on GRDB alone.

Oh hi -- Thanks for your work! Just finished replacing SwiftData in an app with GRDB and it was a pleasure to use.

After struggling with some issues in SwiftData, GRDB really hits the nail on the head in terms of providing a solid dev experience for the common cases, but allowing you to drop into the more advanced features when you need them.

You're welcome :)

Used GRDB many times in a previous life, thank you very much for your work

Sorry about that, just opened a PR to update the readme and docs landing page! Previously it was very clear that GRDB was being used under the hood, but now we will say it explicitly.

Thank you for GRDB! I am using it in a project now and it’s been great. About the benchmarks in this repo though, how can SQLiteData be faster if it uses GRDB under the hood? Are they doing something inefficient or are they bypassing GRDB in some way?

GRDB appears to encourage Codable which is very slow. It does not require it though and there are some alternatives, some of which are also slow. ("Slow" can still mean "fast enough", depending on the user.)

SQLiteData uses this library which uses Swift macros to generate performant interfaces to schema statically at build time: https://github.com/pointfreeco/swift-structured-queries

The alternative I've seen for doing this with GRDB seemed more cumbersome and lacks community adoption: https://github.com/Jasperav/GRDB-ORM You must define schema and queries in an unusual external file that a Rust tool transforms for you.

There is also this library which does not use GRDB but takes a similar approach to SQLiteData though you have to run a program that generates the bindings outside of your normal build: https://lighter-swift.github.io/documentation/lighter/perfor...

Now I understand. Thanks!

Yes. GRDB encourages Codable because the user can profit from the code generated by the compiler, and this implies that database values are accessed by column name, on top of the Codable runtime, and those layers have a high cost. When necessary it is possible to access database values by position, and in this case GRDB achieves speed of light (performance nearly identical as raw SQLite).

From my understanding this is a sample of the database values by position approach: https://github.com/Lighter-swift/PerformanceTestSuite/blob/m...

That approach benchmarks at 2.2x the duration of StructuredQueries (45% as fast): https://github.com/Lighter-swift/PerformanceTestSuite/blob/8...

18.819s vs 8.511s

So it appears that there is lightning-fast and lighting-faster.

Of course aside from comparing the dev ergonomics (138 vs 33 lines for the respective benchmarks), either may be fast enough depending on the use case.

BTW I did also see some discussion in swift-evolution about a future replacement for Codable but haven't tracked its progress. I hope they do because Codable is very convenient but tragically slow.

Yes, GRDB even w/ manual help is obviously not as fast raw SQLite. As much respect I have for the author "performance nearly identical as raw SQLite" is incorrect. Lighter also achieves some of the performance characteristics by avoiding allocations for bound parameters (such being statically generated). I didn't look into SharingGRDB yet, but it seems like those macros could accomplish similar performance, the other way around (Lighter works on the SQLite DB schema). What I'm not entirely sure of yet is why it even sits on top of GRDB in the first place, instead of just doing the SQLite parts itself. Marketing I suppose.

> Marketing I suppose.

Nope. And not sure where you get that idea. This release even involved a rename away from including "GRDB."

When 0.1 of the library was released, it was a simple adapter between our Sharing library and GRDB, thus the name SharingGRDB. As our needs grew, the tool evolved significantly, and both the Sharing component and GRDB have become more of an implementation detail. In the future we will consider supporting any SQLite adapter, even your libraries ;)

GRDB is an invaluable tool to me and, IMO, to the Swift community — thank you for open-sourcing your countless hours of work and expertise!!

> though it doesn't have CloudKit support out of the box

There is no CloudKit support for GRDB available anywhere except for via this SQLiteData package

"Harmony" was an experiment in adding it to GRDB on iOS 17+ but reports claim that it's broken. SQLiteChangesetSync is another earlier experiment that is unmaintained

I don't think it is an experiment, AFAIK it is used in a widely used application by the author.

The abstractions you build your code on top of can be very costly, and a lot of that cost is due later.

You need to carefully weight the costs of abstractions you adopt. That goes double (or triple or quadruple) for data access, which is typically the central point of an app.

When sqlite is involved, I'm really doubting an ORM is worth it in the great majority of cases.

That's because sqlite is robust, mature, full-featured, very well documented, has demonstrated long-term viability, has a commitment to backwards compatibility and support, etc. I think it makes the most sense to implement an application-level data-access layer and have it use sqlite as directly as possible.

What sqlite doesn't do that an app needs is dealing with Swift types. I.e., the grunt work of the sqlite3_bind_* and sqlite3_column_* calls to transform Swift values to and from sqlite3 values. (And to a lesser extent, various Swift-isms for quality of life.) But you can have that without the rather more intrusive footprint -- and therefore much higher cost -- of an ORM.

How else will you support CloudKit with SQLite? Personally CloudKit is a requirement as my customers value having their personal data inside their own Apple account rather than storing it on my servers. I also rely on CloudKit to avoid the cost and on-call burden of operating a realtime sync service.

Re: ORM, the SwiftData-like ORM part of SQLiteData is actually completely optional. You can write SQL statements directly via #sql: https://swiftpackageindex.com/pointfreeco/swift-structured-q... You can wrap usage of this in your own method so that you can swap out SQLiteData in the future without lock-in.

Re: Swift types, you may roll your own mapping, but performance is not trivial. Here is a benchmark from SQLiteData:

Orders.fetchAll setup rampup duration

   SQLite (generated by Enlighter 1.4.10) 0        0.144    7.183

   Lighter (1.4.10)                       0        0.164    8.059

   SQLiteData (1.0.0)                     0        0.172    8.511

   GRDB (7.4.1, manual decoding)          0        0.376    18.819

   SQLite.swift (0.15.3, manual decoding) 0        0.564    27.994

   SQLite.swift (0.15.3, Codable)         0        0.863    43.261

   GRDB (7.4.1, Codable)                  0.002    1.07     53.326

I found Enlighter and Lighter more intrusive to adopt, and the other open source solutions far slower (too slow for my needs, where my users have hundreds of thousands or millions of rows within the iOS apps)

Are you suggesting SQLiteData is an ORM, or SwiftData? SwiftData (and CoreData) are certainly ORMs, but SQLiteData is not. It's simply a collection of tools on top of SQLite that provide similar functionality to SwiftData, but you always have direct access to SQLite.

The things you say that SQLite doesn't do is exactly what SQLiteData provides (Swift-friendly bindings for encoding and decoding data from SQLite), and more. There's a footprint, as there is with any library, but there is no ORM level of abstraction here.

Well, it has things like @Table and wants you to write SQL using Swift syntax. It's good that it lets you bypass this, but it encourages using the wrappers so I think that's going to be most of the usage. Also, if you're going to mostly bypass, then why incur the costs of adopting? The main bypass I saw was #sql, but while you can write SQL, it's a substantially different API than sqlite's C API to do the same.

One problem with data access wrappers is that what's both above it and below it have strong app concerns. The developer needs to understand and control what's below it, so an intermediate abstraction gets in the way. That is, in addition to understanding the lower-level, they also have to understand the intermediate abstraction, and how it maps to the lower level. So it's really best if the API surface is minimal.

What I would want is an API where the core is something like:

bindParameters(pStmt, anEncodableThing) readRowColumns(pStmt, ADecodableThing.Type) -> ADecodableThing

where pStmt would be as minimal a wrapper around a sqlite statement as is feasible (maybe even a pointer to a sqlite3_stmt directly?). There might be a minimal (non-existent?) wrapper around the sqlite3 connection too. (The sqlite pointer should be a public member of any wrapper, so you can call any sqlite3_ functions you want).

(I'd want some convenience methods too, that combine bindParameters and readRowColumns with preparing a statement and stepping through it, like sqlite's exec.)

Now, I know this doesn't address CloudKit sync at all, but I think a similarity minimal type-focused binding is best there too. It has nowhere near the 5-star API that sqlite has so there's a better argument for wrapping it, but sync tends to quickly accumulate app-level concerns when it comes to the exact details of sync conflicts, so you might as well keep that at app level. I think there maybe a set of composable convenience methods might do it, to handle the pain points without hiding the details you'll need access to.

Anyway, my point is NOT that sqlitedata is bad, it's that people should be really careful about taking on the costs that it has, and consider whether it will ultimately cause more problems than it solves. Meanwhile, sqlite is world-class -- you want to wrap it as little as possible.

World-class doesn’t help me if there is no off the shelf or easily made solution to CloudKit sync. Which hasn’t existed until SQLiteData.

(Previously I rolled my own sync with RealmSwift but that’s now dead.)

Without that, my only alternatives are SwiftData or CoreData, with severe downsides.

I find "Fast lightweight replacement for SwiftData that can target all the way back to iOS 13" (when SwiftUI came out I think) to be such a hard sell. How many businesses are there that must absolutely (1) run their apps on iOS 13; (2) use declarative syntax for data persistence; and (3) are profitable enough to invest time and effort on learning a third-party library for a really old OS? It sounds like an overly specific use case.

In general though, I'm not sure who finds PointFree's work a net positive investment of resources, but the company seems to be in the business of reinventing the wheel and locking you in so that you'll pay for support. Meanwhile, Apple's own SDKs are free. And with Apple's history of source-breaking changes over major platform updates, plus given how even huge libraries/tools like Alamofire, Realm, RxSwift, Cocoapods eventually succumbed to oblivion, I can't think of why an Apple developer with any modicum of discernment would choose PointFree's tools over Apple's own--unless they are themselves caught by the allure of reinventing the wheel.

The purpose is not iOS 13 (they perhaps should not place that in the lede). But even iOS 17 is hard to use swiftdata on, given indexes were not even added until 18. And many deploy still to iOS 17 or even 16. I don’t know why you see support for that as a suspicious negative.

But for me the draw is:

- Can use this outside SwiftUI views, unlike SwiftData

- Can adopt future improvements without waiting on annual release cycle + several more years until I can sunset older OS compatibility

- Performance: SwiftData has severe performance issues that are manageable with SQLite

- Cross-platform: SQLiteData depends on GRDB which is perhaps close to merging support for Android (and already supports Linux), at least as an experimental trait: https://github.com/groue/GRDB.swift/pull/1708 By using it with https://skip.tools I can deploy the same SwiftUI codebase to both iOS and Android. And with SwiftCrossUI I can target Windows, Linux too while reusing the same model/persistence layer code. Perhaps this will work in WASM too.

And lastly, many apps already choose to use SQLite. You can see several commenters here who use GRDB in their projects. This library adds CloudKit sync to that which has previously not existed outside of broken experiments or proprietary in-house solutions. The CloudKit capability is more relevant than a decision for whether to use declarative syntax.

Now that RealmSwift is dead, there are to my knowledge zero other alternatives to this new library and SwiftData/CoreData for a CloudKit-synced DB.

Re: pointfreeco I’m unfamiliar with their other work and have no interest in TCA. This library has no dependency on TCA and no lock in on the esoteric or paid parts of their ecosystem that I can see. Most of what it does is provided by GRDB, which has some level of community support and production adoption even from within Apple’s own frameworks (Apple ships GRDB inside of iOS & macOS!). Your comments there seem like fear-mongering and irrelevant to this project. I suggest evaluating this independent of any perceived baggage from their brand’s other work.

I think you're missing the point. iOS 13 is not the value proposition of the library, it's simply one small feature of comparison of many and isn't highlighted beyond a simple mention. The library provides just as much value if you are starting a new app today and choose to target iOS 26+.

> And with Apple's history of source-breaking changes over major platform updates, plus given how even huge libraries/tools like Alamofire, Realm, RxSwift, Cocoapods eventually succumbed to oblivion, I can't think of why an Apple developer with any modicum of discernment would choose PointFree's tools over Apple's own--unless they are themselves caught by the allure of reinventing the wheel.

Isn't this just a blackpilled take in general? You're complaining that Apple software breaks, that other third party software is discontinued, and this somehow leads to the conclusion to avoid this library?

If you think that through, the answer is: of course! Take for example the move from Swift 1 to 2 (an extreme example but illustrates the point).

If you used a third-party lib written in Swift 1 and you had to move to Swift 2 because the next version of iOS requires the latter. Then you’ll have to wait for the lib developer to publish a version of their third-party lib for Swift 2 before you can publish your app. That’s the same kind of risk that you’re exposed to with source-breaking changes.

Admittedly, source-breaking changes have gotten less frequent in Apple’s major tooling updates, but the right mindset when developing for walled gardens like Apple is that it will happen again.

You appear to have unwittingly made an argument for why to use this library over SwiftData - here you can control your own destiny instead of waiting for bugs to be fixed in the OS frameworks and customers to update.

I'm half of Point-Free and can answer your questions.

The fact that our library deploys back to iOS 13 is just our way of showing we care deeply about back deploying these tools so that anyone can use them. Certainly no one is deploying iOS 13 apps these days, but people definitely are deploying iOS 16 apps (and may be for another year or two), which has no access to SwiftData.

And investing time into learning a third party library is just the name of the game when one feels that the tools Apple provides do not suit their needs. Luckily our library mimics many of the patterns that one is used to with SwiftUI and SwiftData, but has the benefits of being based on SQLite, is open source, and not a proprietary technology.

And I'm not sure what wheel we are reinventing here. We feel most of our libraries are specifically filling holes that Apple has left open. We also don't do a ton of paid support with companies, but we certainly do answer dozens of questions on Slack, Twitter, GitHub discussion, etc, every day. All for free.

This library is basically Swift/Core Data, just portable farther back in time. It’s not hard to point out where the reinventing the wheel is.

> learning a third party library is just the name of the game when one feels that the tools Apple provides do not suit their needs

Or you can just wait a couple more platform updates instead of always being on the bleeding edge. Apple’s frameworks might not be perfect, but they’re quite exhaustive, and libraries that aim to fill in those gaps are offering diminishing marginal returns that would be unnecessary in 1-2 WWDCs.

I’m sure there are apps small enough to plan for short time horizons, but for companies who plan for their apps to run for half a decade or more, this is the sort of library that you’ll soon mark as tech debt and then refactor for removal when its approaches eventually make it to Apple’s own SDKs. The historical precedent is there.

> open source, and not a proprietary technology

Yea those are great virtues elsewhere but they stop becoming priorities the moment you pay for the license to develop for a walled garden and agree to its T&Cs

> Or you can just wait a couple more platform updates instead of always being on the bleeding edge.

So your advice is to use CoreData until 2028 or so until we can terminate iOS 17 users and start using WWDC24 SwiftData.

And then maybe a few years from now (meaning usable by 2032) Apple fills SwiftData's gaps that SQLiteData meets today (ability to use it in background fetch tasks, scalable performance, usage outside of SwiftUI views, fast JSONB storage & querying, FTS) or has the capability to meet in the near future (sharing, public database etc.).

That's neither compelling nor industry standard, and is why so many businesses haven't adopted SwiftData and have moved on from CoreData.

It's the same reason PSPDFKit built up to a 9 figures business - no one wants to wait around for years and years for Apple to maybe incrementally improve PDFKit and also not provide any support, developer relations, or ability to fix a bug yourself.

What industry standard are you talking about? We literally have large financial institutions making more money than you running their systems on freaking COBOL. It _is_ common in the industry for organizations to hold off on newer technology because refactoring to the coolest new tech is too costly. Even Cocoa frameworks aren’t obsolete just yet, and it’s unlikely to be so for a long time.

Everything you just said is a dishonest excuse so that you can play with a new library using your company’s money and while letting them incur the additional risk of breakage or tech debt. Nothing is stopping you today from using Core Data to build an observable model or to partially adopt SwiftData just in the functionality that it can sufficiently cover.

No need for personal attacks. I don’t work for anyone. And you don’t know or understand my requirements to make such false claims.

If you think that this library is "basically Swift/Core Data", you have to learn more about "Swift/Core Data" and what it does, and why and how.

There’s the cheap personal attack

It persists data to disk in a manner that is easy to query. It allows observing changes to the data so that features update when necessary. It handles graphs of objects just fine thanks to recursive CTE's in SQLite. With a little bit of work it can handle historical edits with undo/redo. It seems to quack like Swift/Core Data to me, but bringing SQLite to the forefront instead of putting needless abstraction layers on top of it.

I just want to say thank you for the great work you guys do! I honestly have no idea why people feel the need to tear you guys down.

Using TCA has been a great tool for me in my toolbox and it’s very clear how much time and effort goes into making these libraries, documentation, videos, online support, etc.

I wish the macro approach to SQLite in Swift was about generating types from a raw query rather than generating table types. I don’t want an ORM. I want to be able to write raw SQL and have the compiler check that SQL for correctness and generate the proper bindings and types for using it. Similar projects like sqlx in rust or safeql in typescript already do this.

> I want to be able to write raw SQL and have the compiler check that SQL for correctness and generate the proper bindings and types for using it.

Our library does allow this a little bit, in addition to a Swift API for building queries. You can write SQL as a string while still using the schema information exposed from the @Table macro. For example, a query to select all reminders lists along with a count of the reminders in each list:

    @Table struct RemindersList {
      let id: UUID
      var title = ""
    }
    @Table struct Reminder {
      let id: UUID
      var title = ""
      var remindersListID: RemindersList.ID
    }

    #sql("""
      SELECT \(RemindersList.title), \(Reminder.id.count())
      FROM \(RemindersList.self)
      LEFT JOIN \(Reminder.self) ON \(RemindersList.id) = \(Reminder.remindersListID)
      GROUP BY \(RemindersList.id)
      """,
      as: (String, Int).self
    )

Oh I missed that on my first read through of the README. Very nice!

Love it! Not an iOS dev, but have built a personal app that I want to publish for macOS too. I built on top of SQLite, but didn’t want to tackle the synchronisation yet. That should solve it.

It employs a „last edit wins“ strategy for conflict resolution, which is fine in my case. Obviously could be too basic for collaborative apps.

You’d be surprised; it depends on the scope of the last write wins register. If you do it granularly attribute by attribute, last write wins is fine for mostly-online mostly-realtime collaborative apps. The exception is (rich) text where you really want CRDT/OT/similar intention preserving merges of concurrent edits.

SQLiteData does do it granularly by attribute

Unrelated, but as a backend dev who recently worked in Swift/SwiftUI to build an app, the iOS developer experience is absolutely horrendous

Heh, let’s see how you like Android and React

React Native is actually quite good at this point, I've worked with it pretty extensively before doing a native app. I prefer it so I don't have to deal with iOS stuff directly

What would you use from RN to achieve CloudKit sync? I haven't seen anything notable.

Many users value iCloud sync because it keeps their data private to their own Apple account, without having to share access to their data with the app developer's servers.

Reply on news.ycombinator.com