Hacker News Reader

Comment by Groxx | Hacker News Reader

I do feel like this largely summarizes as "we built our own sqlite + raft replication", yeah. But without sqlite's battle-tested reliability or the ability to efficiently offload memory back to disk.

So, basically, https://litestream.io/ . But perhaps faster switching thanks to an explicit Raft setup? I'm not a litestream user so I'm not sure about the subtleties, but it sounds awfully similar.

That overly-simplified summary aside, I quite like the idea and I think the post does a pretty good job of selling the concept. For a lot of systems it'll scale more than well enough to handle most or all of your business even if you become abnormally successful, and the performance will be absurdly good compared to almost anything else.

View on HN

They basically only save on serialization & deserialization at query time, which I would consider an infinitesimal saving in the vast majority of use cases. They claim to be able to build some magical index that's not possible with existing disk-based databases (I didn't read the linked blog post). They lose access to a nice query language and entire ecosystems of tools and domain knowledge.

I fail to see how this little bit of saving justifies all the complexity for run-of-the-mill web services that fit on one or a few servers as described in the article. The context isn't large scale services where 1ms/request saving translates to $$$, and the proposal doesn't (vertically) scale anyway.

One thing I forgot to mention: if you use a not-in-process RDBMS on the same machine you also incur some socket overhead. But that’s also small.

You should probably RTFA before making broad assumptions on their solution and how it works. Most of what you wrote is both incorrect and addressed in the article.

Telling people to RTFA is against site guidelines. And I read the entire article before making this comment. If you think I’m wrong, you reply with what’s wrong, not some useless “you’re wrong, RTFA”.

The only thing in my comment that’s not directly based on the article is a handwavy 1ms/request saving estimate, and since they don’t provide any measurement, it’s anyone’s guess.

Is telling the people to RTFA against the guidelines?

The guideline specifically advises to do what GP did: Instead of commenting whether or not someone read the article, to tell them that article answers their questions.

Rqlite would be a better comparison. It is actually SQLite + raft

https://github.com/rqlite/rqlite

rqlite author here, happy to answer any questions.

So some dumb questions if you don’t mind

- In GitHub readme you mention etcd / consul. Is rqlite suitable for transaction processing as well ?

- I am imagining a dirt simple load balancer over two web servers. They are a crud app backed onto a database. What is the disadvantages of putting rqlite on each server compared to say having a third backend database.

It depends on what kind of transaction support you want. If your transactions need to span rqlite API requests then no, rqlite doesn't support that (due to the stateless nature of HTTP requests). That sort of thing could be developed, but it's substantial work. I have some design ideas, it may arrive in the future.

If you need to ensure that a given API request (which can contain multiple SQL statements) is atomically processed (all SQL statements succeed or none do) that is supported however [1]. That's why I think of rqlite as closer to the kind of use cases that etcd and Consul support, rather than something like Postgres -- though some people have replaced their use of Postgres with rqlite! [2]

[1] https://rqlite.io/docs/api/api/#transactions

[2] https://www.replicated.com/blog/app-manager-with-rqlite

Thank you - so my takeaway is that rqlite is well suited for distributed “publishing” of data ala etcd, but it is possible to use it as a Postgres replacement - thank you I will give it a go

As for your second question, I don't think you'd benefit much from than that, for two reasons: - rqlite is a Raft based system, with quorum requirements. Running 2-node systems don't make much sense. [1] - Secondly, all writes go to the Raft leader (rqlite makes sure this happens transparently if you don't initially contact the Leader node [2]). A load balancer, in this case, isn't going to allow you to "spread load". What is load balancer is useful for when it comes to rqlite is making life simpler for clients -- they just hit the load balancer, and it will find some rqlite node to handle the request (redirecting to the Leader if needed).

[1] https://rqlite.io/docs/clustering/general-guidelines/#cluste...

[2] https://rqlite.io/docs/faq/#can-any-node-execute-a-write-req...

I'll throw in a "ehh... sorta" though rqlite is quite neat and very much worth considering.

The main caveat here is that rqlite is an out-of-process database, which you communicate with over http. That puts it on similar grounds as e.g. postgres, just significantly lighter weight, and somewhat biased in favor of running it locally on every machine that needs the data.

So minimum read latency is likely much lower than postgres, but it's still noticeable when compared to in-process stuff, and you lose other benefits of in-process sqlite, like trivial extensibility.

There's dqlite, which is a in-process sqlite with raft.

https://dqlite.io/

SQlite doesn't do Raft. There isn't any simple way to do replicated SQlite. (In fact, writing your own database is probably the simplest way currently, if SQlite+Raft is actually what you want.)

What about rqlite?

Reply on news.ycombinator.com