Chiselstore: an embeddable, distributed SQLite for Rust | Hacker News Reader

Chiselstore: an embeddable, distributed SQLite for Rust

github.com

58 pointspatrickdevivo21 comments

I've looked at the https://github.com/chiselstrike/chiselstore/blob/main/proto/... - and calling the service "RPC" is not great if this is going to be bundled with other gRPC services - why not call it "chiselStore" or something more concrete?

on a 2nd note, each method should have it's own Request Response, and ideally they should be suffixed same way. Returning same proto for multiple methods might break future compatibility - e.g. what if AppendEntries needs to return somehing else than Void? - ideally you make it from the start to return an empty AppendEntriesResponse, and then you can extend that proto.

https://developers.google.com/protocol-buffers/docs/proto3#u...

(edit again, also the package should not be called just "proto")

They can replace Void with AppendEntriesResponse anytime they want and that won't break anything since the change is wire compatible.

Didn't know that! (Good to know, and thanks!)

Am I correct that this orders and makes SQL statements durable via Raft and then executes them serially against a local SQLite database?

How would you use this to perform a transaction, or at minimum a compare-and-swap operation against a row? There doesn't seem to be any binding between a client and a connection for you to implement transactions here.

I haven't read much into it, but since it's Raft, it's most certainly executing all writes/updates as well as consistent reads against a single leader node.

All changes to the database would then be propagated through Raft using a commit log, which would be read and accepted by follower nodes. This is also reinforced by the fact, that nondeterministic operations are supported.

Thus, everything around transactions and CAS's will work the same as it normally does, as it executes on the leader.

Yes, that's exactly how rqlite works. If I ever get around to adding proper transaction support it will have to overcome the challenge you identify. I believe it can be done, but it will require significant changes to the current design. But an explicit goal for rqlite has always been to do a certain set of things very well, before doing something new. So I'm cautious.

Systems like Chiselstore and dqlite operate at a different architectural layer, so can offer transaction support. But require more programming to use. It's a trade off.

Same idea as https://github.com/rqlite/rqlite I presume but in rust vs go?

And likewise, Dqlite for the C folks in the crowd:

https://dqlite.io/

[Edit] The only thing that'd be different for Rust and/or C is the avoidance of "stop-the-world" GC pauses, which affect Java more, but are still present in Golang.

dqlite has 2 other advantages over rqlite: it supports transactions, and more importantly (IMHO), it works with Go's native sql.DB handler.

Yeah, I need to get around to creating a Go sql.DB layer. Been too busy adding to the database itself. :-) Soon!

Please do! That was the deal breaker when I was trying to migrate an internal project to SQLite/rqlite (couldn't slap an ORM on it).

Have you used this? Is it good? Having a distributed database seems like a great thing, but it kind of rubs me the wrong way that that database is SQLite; I associate it more with lightweight/in-process stuff, which is the opposite of distributed.

Sure, but rqlite tries to give you the best of both worlds -- a distributed database that is also lightweight and super-easy to run.

I came here to mention rqlite too. It's awesome, and non-voting nodes let you replicate out to many, many nodes w/o much performance issues.

Cheers, glad you like it -- great to hear.

rqlite author here, happy to answer any questions about it. It seems some folks have questions about its design, and the design of systems that combine SQLite and Raft in general. Checking out the rqlite design docs might be helpful:

https://github.com/rqlite/rqlite/blob/master/DOC/DESIGN.md

rqlite is not embeddable, making it harder to colocate data with computation. This post explains it: https://glaubercosta-11125.medium.com/728187331f53

The announcement here talks more about the context:

https://glaubercosta-11125.medium.com/728187331f53

It also mentions rqlite and dqlite.

How's the security and security defaults and are these documented somewhere? Are the "cargo run" examples safe to run as is in a networked environment without random people on the internet pwning your cluster? Is there confidentiality & authentication in the inter node communications?

Isn't the whole point of SQLite to not be distributed?

Not really, just that you can use it in-process with little to no configuration. But a distributed system using sqlite as the storage seems useful in its own right.

Reply on news.ycombinator.com