Hacker News Reader

Comment by otoolep | Hacker News Reader

rqlite author here, happy to answer any questions. Many thanks to Nienke Eijsvogel, Ruben van Baarle and Daan de Graaf for executing this testing.

https://github.com/rqlite/rqlite

View on HN

Not really related to TFA, but I had a look at the python driver documentation and I was very surprised by this statement [0]:

> Limitations

> Transactions are not supported

Is there something I am badly misreading? How come is a so fundamental feature not supported? Maybe it is just a limitation in the python driver?

[0] https://github.com/rqlite/pyrqlite#limitations=

No, it's due to rqlite.

rqlite does support a form of transactions -- you can send a set of SQLite statements, and either all will be successful, or none will be. You can even send a BEGIN statement, and a SQLite transaction will be opened (and you can COMMIT later). However, due to the nature of Raft, restarting the cluster in the middle of an open transaction will not clear the transaction -- and most folks probably don't expect this. That's why I need to think more before advocating use of traditional transactions with rqlite -- distributed transaction functionality is easy to get wrong, so requires careful thought. You can use SQLite transactions, but you must be careful. That's why drivers don't tend to support it yet.

https://github.com/rqlite/rqlite/blob/master/DOC/DATA_API.md...

Taken from the readme:

> the behavior of a cluster if it fails while such a manually-controlled transaction is not yet defined

Do you envision manually-controlled transactions working seamlessly one day? If not, have you thought about what guarantees are feasible/you hope to provide?

Yes, I have a design in mind, but it's a fair amount of work. It will require a new API too -- and API that allows clients to explicitly create a new connection, create a transaction on it, and close that transaction when finished.

I there a go client other than gorqlite which doesn't seem to actively maintained.

Would it be a better compromise to support transactions as long as they were sent as a single block?

Not sure if I follow you, but that's exactly what rqlite does support.

https://github.com/rqlite/rqlite/blob/master/DOC/BULK.md

Sorry, I meant from a marketing perspective, saying "transactions are only supported in batch mode" rather than saying they aren't supported.

MySQL didn’t have transactions for years! A transaction layer can be implemented by adding a transaction table, a transaction column to each other table, and joining the two in your queries.

  where table.transaction_id = transaction.id and transaction.committed = true

You would begin a transaction by inserting a new row and then updating it to committed when done.

With some metaprogramming you may be able to do this all transparently.

How would this provide read / write isolation? Or rollback in case of failure(power outage)?

As an exercise for the reader. :)

Both are possible and have been done before.

Very cool project putting together raft and sqlite. Definitely seems like it's biggest benefit would be super simple operation when you also really need the distributed behaviors. Is that accurate? What do you guys use it for and where do you think the sweet spot is for its application?

Yes, simplicity of operation is a key goal of rqlite. To quote from the FAQ:

rqlite is very simple to deploy, run, and manage -- in fact, simplicity-of-operation is a key design goal. It's also lightweight and easy to query. It's a single binary you can drop anywhere on a machine, and just start it, which makes it very convenient. It takes literally seconds to configure and form a cluster, which provides you with fault-tolerance and high-availability. With rqlite you have complete control over your database infrastructure, and the data it stores.

https://github.com/rqlite/rqlite/blob/master/DOC/FAQ.md#why-...

Another commenter mentioned automatic sharding to which you replied "no". But short of that what do you think about use cases where the database is "naturally" very sharded, say where every user has a separate database? With such a design the operator might want to scale rqlite down to zero instances when a db is unused for a while and start it back up quickly on demand.

Have you thought about how rqlite could fit into this design space?

That seems better solved outside of rqlite in the application layer and inetd or k8s.

I have considered it, but it would introduce a very large amount of complexity -- and make rqlite much more complicated to operate. Adding this type of functionality would push mean rqlite would no longer be a worthwhile system to use. It would do something (sharding) probably no better than other systems, but no longer be trivial to operate. rqlite does something somewhat narrow, but does that very well (though I would say that :-) )

I have to say, I like your product thinking. I agree with you.

Reply on news.ycombinator.com