Hacker News Reader

Comment by randomwebdev | Hacker News Reader

(Disclaimer: I'm PocketBase author)

It is nice to see more backends utilizing SQLite. The benchmarks and the Comparisions section also seem well done.

Just a nitpick - list the versions of the tested platforms.

Based on your benchmarks repo it looks like that the tests were done against PocketBase < v0.23 but note that PocketBase v0.23+ (especially with the Create API rule dry submit removal in v0.24+) has introduced significant changes and performance improvements - ~4x times in high concurrent scenarios in our own benchmarks[0] (if you want to retest it note that the CGO driver is no longer loaded by default and will have to be registered manually; see the example "db_cgo.go" in the PocketBase benchmarks repo or in the "Custom SQLite driver" docs[1]).

[0]: https://github.com/pocketbase/benchmarks

[1]: https://pocketbase.io/docs/go-overview/#custom-sqlite-driver

View on HN

- It is nice to see more backends utilizing SQLite.

Hey thanks for chiming it. Huge fan of PocketBase, has been a major inspiration :applause:. For anyone driving by, certainly a more mature product.

- Based on your benchmarks repo it looks like that the tests were done against PocketBase < v0.23 but note that PocketBase v0.23+ (especially with the Create API rule dry submit removal in v0.24+) has introduced significant changes and performance improvements - ~4x times in high concurrent scenarios in our own benchmarks[0] (if you want to retest it note that the CGO driver is no longer loaded by default and will have to be registered manually; see the example "db_cgo.go" in the PocketBase benchmarks repo or in the "Custom SQLite driver" docs[1]).

You're right. I did run v0.22.21, which simply was current when I ran the benchmarks first. I absolutely will add the information, rerun, and thanks for the pointers. Glad to hear you got such a boost :clap:

I'm happy to report that v0.25.0 already w/o CGO driver (fighting it right now) and GOOS=linux CGO_ENABLED=0 GOAMD64=v4 improved about 35% from 61.7s per 100k inserts to about 40s :clap:

I still wanna get the mattn/go-sqlite3 driver to work and it's getting a bit late here for writing coherent text... I'll update the benchmarks ASAP

Back from the dead. I also now vaguely remember that there was some hiatus (probably between v0.22 and v0.23) where you took some time to rewrite the DB layer to enable custom DB drivers.

Anyway, w/o fog I managed to run with mattn/go-sqlite3. I'm not sure this is expected but it didn't seem to make much of a difference with my setup (For transparency, I do recently have some issues with repeatability likely due to btrfs). I'm certainly not seeing 4x but around 30-35%, which is still very impressive!

> I'm not sure this is expected but it didn't seem to make much of a difference with my setup

I expect it to be faster but it is also possible that maybe in your specific collection and execution scenario it somehow perform worst, or at least not the same as in my benchmarks, I'm not sure.

The ~4x mentioned speed up is based on the tests for Hetzner CAX41 with the CGO driver when creating 50k records with 500 concurrent clients:

Old (58.409064001s): https://github.com/pocketbase/benchmarks/blob/54140be5fb0102...

New (13.580098262s): https://github.com/pocketbase/benchmarks/blob/7df0466ac9bd62...

---

Edit: Based on your benchmarks repo (https://github.com/trailbaseio/trailbase-benchmark/tree/main...) note that compiling PocketBase with `CGO_ENABLED=1` is no longer enough to kick in the github.com/mattn/go-sqlite3 driver and you need to enable it manually as shown in the db_cgo.go in our benchmarks OR like in https://pocketbase.io/docs/go-overview/#github-commattngo-sq... (the reason for this change was to avoid the "multiple definition" linker errors).

> Edit: Based on your benchmarks repo (https://github.com/trailbaseio/trailbase-benchmark/tree/main...) note that compiling PocketBase with `CGO_ENABLED=1` is no longer enough to kick in the github.com/mattn/go-sqlite3 driver and you need to enable it manually as shown in the db_cgo.go in our benchmarks OR like in https://pocketbase.io/docs/go-overview/#github-commattngo-sq... (the reason for this change was to avoid the "multiple definition" linker errors).

FWIW, I did this locally (also happy to fork PB and check it in, certainly aids transparency). I've already updated the numbers in the benchmark doc but will continue to try squeeze more out of it.

As to 4x, my concurrency levels are significantly lower which could certainly explain it. Is PB juggling multiple connections increasing write lock congestion or are you serializing access, e.g. via a worker thread? Also happy to chat more (feel free to send me an email), I certainly want PB to have the best possible representation and maybe we can even speed things up on both sides

> As to 4x, my concurrency levels are significantly lower which could certainly explain it.

Yes I noticed that too so that it is very likely the reason.

In all cases my initial comment wasn't intended to "dispute", argue or anything like that, so please don't feel obligated to waste time updating the benchmarks. They are valid as they are!

My initial comment was more of a note/suggestion to simply list the used versions of the tested platforms (not just for PocketBase) because often they change over time and the shown results could be misleading if someone stumble on the article 2 years later for example.

> My initial comment was more of a note/suggestion to simply list the used versions of the tested platforms (not just for PocketBase) because often they change over time and the shown results could be misleading if someone stumble on it 2 years later for example.

FWIW, it never felt like a dispute and very much agree with your suggestion. I'm also just trying to do a decent enough job, both with the benchmarks and TrailBase itself. Either way, my offer to keep an open channel stands in case you want to share experiences or are in desperate need for a beverage :)

Reply on news.ycombinator.com