Hacker News Reader

Comment by southerntofu | Hacker News Reader

> is it a series of messages? Or is it something more advanced; with built in conflict resolution or other P2P-y features?

From the client's perspective, matrix is just a basic stream of events (like XMPP). The magic happens on the server side where a room is replicated across servers and conflict-resolution voodoo is applied. That's also why synapse server is really really resource-intensive (gigabytes of RAM) compared to equivalent-in-features (apart from decentralized rooms) XMPP/ActivityPub servers ; dendrite is fairing better but i haven't benchmarked it yet.

View on HN

this isn’t really correct (but is a common misunderstanding). from the client’s perspective, a matrix room is a persistent timeline of events which you can long-poll (/sync) and navigate forwards and backwards (/messages). Events can also update key/value state associated with a room (eg that room’s name, topic, avatar, membership, or any other metadata), which you query via /state. This is not a stream of events like XMPP but a replicated DAG of events.

Synapse’s RAM usage is a) way better these days, b) not due primarily to the merge resolution algorithm (state res), but due to slapping in-memory caches everywhere rather than being smarter on the db schema. Dendrite does 5x better by being smarter in the schema, and Conduit looks to be even better via cunning use of using key/value dbs for persistence (sled).

> This is not a stream of events like XMPP but a replicated DAG of events.

Thanks for the clarification. My point was that building something client-side for matrix is pretty straightforward, in case that was not clear.

> not due primarily to the merge resolution algorithm (state res), but due to slapping in-memory caches

Good to know. conduit doesn't quite look to be ready, but would you recommend using dendrite in production to serve a few users? i guess i'll give it a go

Dendrite is still beta but is usable (as hummingbard is demonstrating). There are some missing features which make it less useful day to day currently, like push notifs, and currently the Dendrite team is off doing P2P and low bandwidth Matrix work. It’ll be back soon enough though.

Thanks for the info. If the only thing that's missing is mobile support (push & low-bandwidth) i'll sure give it a go.

For the key-value storage, would it make sense to use it as a "large" KV database?

Eg, for sake of simplicity, imagine you're storing Git in Matrix (similar to what i'm doing, if i use Matrix). I've got two primary concepts:

1. A mutable set of pointers.

2. A series of key values, where the keys are content addresses, and the values are a byte array.

My initial assumption would be to use the messages to post all newly created blocks and changes of pointers. Eg, Send a message with a new content address and the bytes. Send a message with a new address for a pointer.

For a Git-like to work in that model, it just needs to slurp up any messages in the datastream. Making sure to also slurp in old messages properly that maybe are out of order (ie, reconnecting after being offline for a while). But being a Git-like, it's a fairly simple model.

With that said, perhaps the metadata storage in Matrix would better suite some of this? I just imagine Matrix isn't intended to host TiBs of `/state` metadata.

Thoughts?

Yeah, Matrix isn't a good fit yet for freeform large KV storage. For instance, there isn't a good way to paginate state yet: you can query values by a given key, or a given type, but not paginate. Also, each KV pair gets fed into the merge resolution algorithm, which can start to get sluggish with more than ~100K events (although it's got way better in Synapse since https://github.com/matrix-org/synapse/blob/master/docs/auth_... landed a few weeks ago).

In the longer term we'd (well, I'd) love to make it work well with freeform KV data (or object graphs! you gotta store your VR scene graphs in Matrix, after all) and act like some kind of crazy global zero-trust CouchDB or Cassandra. But right now the KV state is meant to be for relatively simple freeform metadata about a room.

> In the longer term we'd (well, I'd) love to make it work well with freeform KV data

That would be great for a decentralized forge based on matrix. This is an area where i think matrix decentralized rooms would shine, as they would effectively prevent DMCA takedowns for software projects.

Reply on news.ycombinator.com