Show HN: Goqite, a persistent message queue Go library built on SQLite | Hacker News Reader

Show HN: Goqite, a persistent message queue Go library built on SQLite

goqite.com

99 pointsmarkusw69 comments

I wanted to build a persistent message queue based on SQLite, because that's what I'm using for my main state anyway. This gives me ACID across state and messaging, which is nice!

I've been inspired by the terminology of AWS SQS for this, but it's obviously much simpler.

Maybe you can use it too. :)

Open articleView on HN

Well peepz, it’s been fun, but I’m off to bed now because sometimes I get sleepy. Ta! :D

(Thank you for all the comments!)

> says they are sleepy

> 30 minutes later they open a new PR to address a comment left by [an HN user](https://news.ycombinator.com/item?id=39673795)

I think this person might like coding

Haha, you got me. :D

In a me too way, I also created something similar - https://github.com/klev-dev/klevdb

It started on top of bbolt, but over time moved a completely custom implementation. The main reason for this was performance, but I also wanted to run a lot of queues, so each one had to be as light as possible. On top it even provides some indexing capabilities, which (when enabled) allow for quick find by key and time.

Interesting, the performance of https://github.com/klev-dev/klevdb is 10x https://github.com/maragudk/goqite so it makes me assume the durability is somewhat lacking. Can you speak to the tradeoffs here around message loss?

I would think that having a small chance of message loss due to writing to an append only log in batches might be a reasonable trade off for many things (if that is how it works).

First off, looking at goqlite benchmarks, we are measuring different things - goqlite is measuring a complete lifecycle of a message, which includes sending, receiving and deleting. The klevdb benchmarks are usually exercising one (at most 2) of these.

In klevdb you can control the fsync yourself - using AutoSync you can ask it to perform fsycn after each Publish, which currently is best effort put on disk. Of course in this case performance suffer dramatically (as you would expected, esp if you don't batch). You can also not use AutoSync and control it yourself (klevdb exposes Sync), deciding what kind of message loss you are prepared to deal with. The benchmarks you see run without AutoSync, to highlight what kind of raw throughput klevdb is capable of.

And yeah, klevdb is basically an append-only log. Of course, you can delete messages from it, but this is done in parallel, where the segment is being copied without the messages you want to delete, then replaced with a single mv system call. Rewriting the head segment (e.g. the one you are writing to) is not always possible (since there could be messages the rewriter haven't seen), but will eventually succeed (once the head segment is no longer a head).

Hope this answers your questions, and if not, happy to expand more. I've been contemplating on implementing transactions on top of klevdb, but haven't quite figured out if I need them or not.

Edit: spelling and grammar

I also built something similar for Python [0]. I decided to go with a separate DB file for the queue, I think it's better for performance but also to avoid mixing the data storage with the message queue (in my case, I assume the message queue is just an implementation detail and not really tied to the rest of the DB schema).

[0]: https://github.com/litements/litequeue

Yeah, that’s a good idea! I was today old when I learned that SQLite can have transactions across separate db files.

Only if you don't use WAL.

Ooooooooh. TIL!

Also did something very similar, but swapped out the storage layer with an embed of leveldb.

Also supports both an HTTP and Redis API.

https://github.com/tomarrell/miniqueue

Interesting that it supports the Redis API as well!

As noted elsewhere here, it was important to me to use SQLite, since that's what I use as the storage layer already. But I've added HTTP adapters for remote clients.

I hadn't thought of using a different remote API, interesting idea. Thank you for the pointer!

This is super cool! I like your method for handling the claiming of queue items. I always end up wishing SQLite had something like Postgres' LISTEN/NOTIFY/SKIP LOCKED but your UPDATE solution is nice and low tech and fitting for SQLite, in a good way!

Thank you, I’m really glad you like it! :)

In the example the user has to create the schema and manually open the SQLite connection, can't you abstract that away?

Not the author, but I imagine most projects already have a way of managing their SQLite schema (e.g. a migration system like go-migrate or goose) that they'd integrate the new table(s) into.

This exactly. If you're using SQLite in your Go project, you most probably already have your sql.DB and schema migrations sorted, so I don't want to get in the way of that. :)

(But I can see how that could be clearer from the example code. I think I'll add that as comments. Thank you for the feedback!)

I've updated the code example to reflect this.

Wouldn't you want to use a different db file for this? I wouldn't want to put this in the same db as the rest of my app, especially considering how simple it is to just open a new file.

If you want to be able to send to the queue in the same transaction as other state changes, it has to be the same file. That’s often a useful property.

SQLite allows transactions involving multiple databases (see https://www.sqlite.org/atomiccommit.html section 5)

Oh! TIL! Thank you for that. :)

SQLite is such a great thing. Amazing value for zero cost both shmeckels and maintenance. It's a shame most startups go for postgres blindly due to "scalability concerns" it feels like a major case of premature optimization.

It really is! I've become a SQLite fanboy. :D Although I can definitely see the value in Postgres as well. But it's usually one of those two.

My beef with SQLite is the weaker typing + not even enforcing foreign keys unless you open the DB in a certain mode.

IMO it should become more opinionated and stricter. I'd honestly pay to buy a license for SQLite with the legacy baggage removed and rewritten to be as strict as PostgreSQL or more. Well... basically embedded PostgreSQL I guess.

I also don’t like those parts, but for an otherwise great project, I’d happily use the parts I do like. Strict tables go a long way for me, and you can make foreign key checks a compile-time enforcement.

Agreed, plus check my other sibling comment: https://news.ycombinator.com/item?id=39681241

Same. I’ve been using it in a project, and the entire time I feel like I’m needing to be twice as careful not to introduce problems.

Yep, I am even considering writing a Rust library that does opinionated opening of SQLite databases with FK enforcing turned on, strict tables and other such goodies, and making sure dates / times and such are properly (de)serialized so we can have some convention inside SQLite itself (f.ex. fully ISO formatted datetimes with or without timezones).

Ha, I’ve started the same thing in Go, but didn’t get that far. https://github.com/maragudk/sqlite

I love Go and SQLite and enjoy using them together. One bummer is that it requires CGO to use and that makes cross compiling a pain. It might be nice to use the modernc variant (via build tags) to make this more portable -- https://gitlab.com/cznic/sqlite

It’s bring-your-own-driver, so you can use a CGo-free one. :) The one in go.mod is only used in tests.

Zig allows cross compilation of cgo

There is a pure Golang SQLite implementation btw, I love that aspect of the Golang ecosystem. Not even Rust has that and I am envious.

Give Rust some time, there's a wee bit of competition going on there ;-)

Oh, absolutely, it's just kind of strange for me that nobody has bothered translating the SQLite C code to Rust, and Golang's community did.

Though I'd guess tools like `c2rust` don't produce idiomatic Rust code and it can be a lot of effort to turn that into proper Rust. That's likely the biggest reason.

I made something extremely similar, but lighter weight awhile ago: https://github.com/chiefnoah/goalpost

It uses bbolt/bolt instead of sqlite. It was for use on shared servers that had questionable uptime.

I haven't touched it in a long time, but it served its purpose well

Yeah, there are a lot of queue libraries like ours. :) It was important to me it was SQLite, since that's what I use as the storage layer already.

Yeah, not trying to detract from your work! At the time when I made mine (like, 6 years ago) there was nothing in the space. Now there's plenty! Nice work!

Didn’t take it that way, but thank you for the clarification. :) And thanks! Likewise. :D

Interesting, this seems to be something that a lot of people tried to implement, it’s the first time I see this many “I did something similar” under a project!

Cool project! Would love to implement something similar in another language and maybe compare performances, even though I guess they would be very similar since there’s gonna be SQLite under the hood.

Yeah, it's not exactly a new idea. I'm actually a bit surprised it's this popular. :D

Thanks! Doing it in another language would be almost trivial, there's really not a lot of code involved. I've focused mostly on making a nice, easy to use API for the common cases. No more, no less.

How is this better than just using SQLite transactions directly and a task queue table?

It has a Go API. ;)

Check the source, that's essentially what it is. But you get this particular design, the tests etc. for free, so if you value your time, maybe this is worth something to you. Maybe not though. Either way, it's fine.

As much as I love SQL I would have thought this would have abstracted away the SQL bit of the queue and just given me high level queue methods to do stuff with. I don’t like this much for that reason.

I don't understand your comment. Abstracting away the SQL is exactly what this library does (and pretty much the only thing): Send, Receive, Delete methods.

Could you clarify?

Well your example on the website contains a Database schema and manually opens a DB connection. Why do I as a library user need to care about that? Just do that internally when creating the queue instance

See this other thread: https://news.ycombinator.com/item?id=39670199

From the example, it appears you still need to setup schema yourself using SQL. This makes the library a leaky abstraction because it only partially hides the SQL.

Exactly this.

There you go! :D https://github.com/maragudk/goqite/pull/31

Wow nice. Quick work.

Looks neat, but how on earth is "Goqite" meant to be pronounced?

Thanks! See the first sentence of the readme. ;)

"goqite (pronounced Go-queue-ite) is…"

You should upcase the Q then. ;)

Hehe. Perhaps.

maybe GoQuette? (like eti-quette or cro-quette, go-quette)

Nah, I've already got the domain. :D

Annoying but I felt exactly the same. I'd advise you to rename the project - it's still early days and the longer it goes on the harder it will be to change - I've felt this pain.

I actually quite like it. :D

I have strong feelings in the opposite direction.

Which is very much your right. Agree to disagree? :D

Ignore feedback at your own risk.

Feedback and opinion are not the same thing.

Also, yes, I will take on that risk. As anyone who has ever been at the receiving side of an issue tracker knows, not all feedback is equal. I appreciate taking the time to provide feedback, though, but I always reserve the right to not act on it.

Is it feedback or opinion then? Your two sentences do not agree.

If 7 people tell you that "go kite" is not a great name for an important database system, I don't know what to tell you other than "consider the feedback".

Cool idea, I can definitely see the appeal of having everything in one database. I would probably also add an "attempts" column (int) so you can discard messages after X attempts.

That’s the “received” column. :) Messages after X attempts aren’t received by the normal receiver method anymore.

Goqlit might've been closer a brand-action name

See the other thread on naming. ;)

Reply on news.ycombinator.com