France Enters the Matrix
lwn.net
lwn.net
There's also an active non-profit organization using FOSS to create alternative, privacy-respecting, solutions to Google products (framasoft[2]). I mention this last one because I haven't come across a similar major undertaking (or similar mindset) in too many other countries.
Now they are evaluating Matrix for their communications needs. I can understand the importance of countries, nowadays, not relying too much on the services of companies tied to a specific country (namely the US) but I wonder what else is different about France? What else are they engaged in re: FOSS, privacy?
Or am I reading too much into this?
[0] https://en.wikipedia.org/wiki/Qwant
So to them it's mostly a matter of choosing which foreign entity to trust and at which cost, between the US, Russia, China, France, and a handful of other states with lower performing but cheaper equipment.
Also, having intertwined economies is supposed to help maintain peace and foster prosperity.
If she would go nuclear on Argentina due to that skirmish and few marines dying in regular acts of war (however sad it is, this is business-as-usual during war), she would have alienated UK to every single country in the world, destroying it on many levels. And also go down in history as one of the most horrible rulers in history of mankind.
Not something any power-and-approval-hungry sociopath (sorry, 'leader of the nation') wants to leave behind.
Gross generalization.
That really isn't realistic. Some allies might have been unhappy with Thatcher for normalising the use of nuclear weapons, and might even have openly criticised the UK. But France, the US, the Commonwealth nations would not just desert an important ally: how would that benefit them? They would have retconned the story and we'd all believe now that Galtieri was the next Hitler and would have subjugated the whole hemisphere if not for Thatcher's brave initiative.
I am pretty sure that especially in weapon systems it is quite common for customers to demand special changes, services and insight.
As for what else we're doing, there's La Quadrature du Net (LQDN)[1] which took part in the recent Google €50m fine, some projects by concerned citizens to keep a watch on politicians[2] and tracking in mobile apps[3], but one of my favourite things comes from the increasing will to release "OpenData" (public data sets from various sources)[4] to let collectivities, regions and smaller units build tools to solve their local problems.
[1] https://www.laquadrature.net/en
[2] https://projetarcadie.com/ (in French)
The result is a lot of top-notch contributors who get involved in FOSS projects in their 20s.
(People seem to think that France doesn't have a tech industry; it very much does but little of it is consumer facing)
(also, exercising nationalism against a colonial power may be to its detriment but also be morally good. This obviously does not apply in the case of France, which itself is a colonial power)
> POLITICS, n. A strife of interests masquerading as a contest of principles. The conduct of public affairs for private advantage.
This is hardly relevant, but I'm using the occasion to share the great piece of satire that's Devil's Dictionary. Here's a few others:
> OCEAN, n. A body of water occupying about two-thirds of a world made for man — who has no gills.
> POLITENESS, n. The most acceptable hypocrisy.
> PERFECTION, n. An imaginary state or quality distinguished from the actual by an element known as excellence; an attribute of the critic.
Edit:
> PREJUDICE, n. A vagrant opinion without visible means of support.
NATIONALISM,n. Discrimination which is based on geography and thus obviously superior to RACISM and SEXISM which are based on biology. Can be further refined to NEPOTISM and ultimately SELFISM.
Now that the big hacker/open source FUD is mostly done for, I think you'll see a lot more countries deploying FOSS tech as a matter of fact an not as an ideological statement or an avant-garde trial.
Regarding privacy: not my field of expertise, but France is far from paradise here, thanks to the terrorist attacks a few years ago.
https://juliareda.eu/2019/02/article-13-worse/
There is lot of private initiatives to make great things, but no state vision to make things right. The only value they see is economical, but not for the people.
To get back on topic, France has really got its FOSS movement moving. There are a TON of Mastodon and Peertube instances located there. It's admirable.
apt install matrix-synapse
and changing one yaml config to suit my needs
i am sure there is a dockerized version
I created 100 accounts for chatterbots. (This is not federated to the main Matrix network -- just local.)
The VM showed some lag when the chat bots connected and started inviting each other to their channels. But once all the invite sending/accepting settled down, it's been pretty smooth. It's randomized, but I think each channel has about 18 or 20 users.
The whole thing is using about 900 MB of RAM, and that's including everything -- CentOS, several containers, the Postgres database, Synapse, nginx, etc.
I run it on some 3usd vps i dont remember 1gb ram i think and use sqlite (its very portable this way). For about 20 people it works great.
It is not exactly efficient but overall still cheaper and better than other solutions ive found.
By default, it sets up Synapse and a bunch of other services around it that are necessary to really make it useful (Postgres instead of the default under-preforming SQLite; mxisd as the identity server; STUN/TURN server for WebRTC calls; riot-web; various bridges; etc)
Footnote: IMO, the barrier for adoption for this kind of services has to be the lowest possible if it wants to gain any kind of traction. Mastodon and Peertube are good examples of this: docker-compose.yml in the project root, and in < 1 hour I got everything working by just setting a couple configuration values that were already in the environments section.
I figure that there is a lot of work going on right now so this might not be in the top priorities, but I really hope Matrix gets there soon.
Keep up the good work!
The Matrix protocol is well worth the read. JSON-LD is a bit of a nightmare to work with, but the gist of it is a solid concept; it has drastically changed my approach to software design.
Could you expand on what you found particularly compelling about the protocol / API, and also how that's changed your software development approach?
From a technical standpoint, there are six specifications (Matrix, Mastodon, XMPP, SIP, SMTP, Microdata) that have gotten the "future of the internet" partially right, in my opinion. Some union of the them, excluding the unfortunate bits, is likely the email/Twitter/Facebook/what-have-you killer. Putting that in consumers' hands and getting them to use it is another story, but we do have the starts of a technical/back-end answer.
For example, you could embed text document edits inside of Matrix messages as Microdata (ignorantly spamming someone aside with thousands of character edits aside). Is your spouse doing their taxes? Send them a chat message with Microdata explaining to the form where the attached document goes.
As a general goalpost, WeChat (having circumvented federation problems) in China is a really great example. You can pay for food at a hotdog stand on the street with chat. Uber merely facilitates communication between taxis and commuters. Whatever the project(s) you are referring to, software usually ends up solving communication problems. Recursively simplify your problem, ultimately realize how communication fits into it, then work your way back up to your initial idea. Federated IM is ludicrously powerful and, especially with graceful degradation, can lead to some sales leads.
In my own view it's a much more robust way of architecting communication: the server(s) should assume that remote parties will be off, and remote parties will come and get what they want at their own pace and time. Maybe they won't receive a few messages immediately but can immediately retrieve multiple when needed. With a stretch you could see a parallel with Kafka: instead of trying to send stuff to consumers as soon as it is produced, just store it, and when clients are up they'll process stuff.
Yes, it does eventual consistency replication, like NNTP, but that's not an incredibly important feature.
Encrypted 1:1 messages should not be stored on servers! They should be sent between clients p2p stile or at least forwarded by servers without storing them.
however, we're adding message retention configuration currently to let you limit how long your messages get persisted for (whether they're encrypted or not): https://github.com/matrix-org/matrix-doc/blob/matthew/msc176...
Storing 1:1 messages (noting that Matrix doesn't differentiate between group chats and 1:1 chats) on a server is beneficial because you might not want to lose your entire private chat history with your family and friends whenever you lose a device (source: this happened to me multiple times with Signal -- which has an awful backup system that I didn't use because it simply doesn't work properly).
Now, there is obviously cases where you don't want the server to keep your messages forever -- and then you might want to have retention policies and so on. But I don't really agree with the security argument, since the most obvious attackers already likely have your messages, and it sacrifices a lot of convenience that most people expect (my partner didn't expect to lose all her Signal messages when her phone died unexpectedly).
From our perspective, the room replication is absolutely critical - it means that the network can partition and the servers can come up and down without the room being disrupted at all. It's the opposite of a MUC, where if the server hosting a MUC is inaccessible or goes down, you're screwed.
https://web.archive.org/web/20090424052832/http://www.redhat...
https://web.archive.org/web/20090520231136/http://www.redhat...
More info (in French): https://cnll.fr/media/etude-2017.pdf (with figures for the FR, DE, UK and rest of EU markets on slide 4).
The main advantages are it is using JSON instead of XML, and HTTP instead of custom protocols so it can work directly from browsers and with lightweight mobile clients. It also has a lot built into the core protocol, where as to get XMPP to do anything useful you will need to use a lot of extensions (and ensure the server and all clients support them).
The only con is it's still a relatively young project. I'm not sure I would want to rely on it as my main form of communication in a company just yet, but for personal usage it is fine.
If you have a dominant xmpp server (like gmail was) that removes xmpp support, its users have the start using other clients to connect to it or start a new social structure from scratch. In essence this makes the xmpp server a dominant player with leverage.
If gmail had used matrix when it pulled the plug all interactions (who knows who, how to route traffic, chat history) would have been replicated at the peers so it would just have removed google’s servers from the equation. The people who had been using it would continue uninterrupted.
I'm not sure about that. If the main matrix server disappeared today that'd be problematic. Yes, you'd have history locally but the same can be said for any native client that stores history locally.
As far as I know there is no way to migrate matrix history now. Is there? (genuinely curious)
This ^^ Currently a large amount of users (I'm told around a half, but I'm not sure) have matrix.org as their home server. So Matrix is effectively building a power-law network with all its drawbacks: when the "power node" of the network goes down the whole network falls apart.
After Synapse 1.0 is released (sometimes next month, IIRC), I'm definitely deploying my own homeserver, and federating it with the rest of the world.
I guess this is the real problem for me - i would love to federate but i can't possibly police or enforce what servers/channels are small enough that my users can join them.
for everything else we’re working on decentralised accounts via MSC1228.
there are also migration scripts and tools to move you between accounts on different servers today (like email).
In both cases, if users have an account on gmail and are using it, and gmail pulls the plug, then they can't use it anymore and can't migrate. The history is still there in the case of matrix, but they can't reuse the same account.
However if another user is homed on another server, they would still keep the knowledge and history on their side: gmail did pull the plug on xmpp, yet people are still using it. Their server still know who their contacts are, what room they're usually in, etc...
Matrix does have the advantage that the chat history is fully replicated on all homeservers so it makes history retrieval much saner, but that's it. Matrix doesn't route traffic (messages go from sender's homeserver to recipients' homeservers, that's it) and doesn't share client's information with the whole world.
That's indeed a subjective view.
> The main advantages are it is using JSON instead of XML, and HTTP instead of custom protocols
The Matrix dev said a day ago that "It's NOTHING to do with HTTP+JSON": https://news.ycombinator.com/item?id=19217412
Also, what custom protocol are you talking about? XMPP works on top of TCP/TLS (or Websocket). Those are the IETF standards. In fact XMPP doesn't require an additional layer of HTTP at all.
> with lightweight mobile clients
Where are they? Anything even remotely comparable battery-wise with Conversations (an XMPP client)?
Not to mention that Matrix has a terrible server implementation which requires several GB of RAM just to join matrix.org room.
> It also has a lot built into the core protocol, where as to get XMPP to do anything useful you will need to use a lot of extensions (and ensure the server and all clients support them).
I fail to see how having a monolith core versus splitted specs has to do anything with code written to support that. Of course, initially every implementation will strive to implement full core (that was in XMPP in 2000s), but when the core evolves you have absolutely no guarantee that every developer will implement all new parts of the core. How is that addressed in Matrix?
Well that's quite easy, since hosting a server is a resource hog, most people are still on matrix.org, which gives it a special status in the ecosystem (even more special than jabber.org has been in the early 2000s fox XMPP). This allows them to "move fast and break stuff" on short notice (a few months), to bring the protocol forward and force people to upgrade. In the federated XMPP ecosystem, they essentially have to come to a community consensus between developers, servers operators and all other relevant actors to write a manifesto or a new spec that says "stop using $thing at $date".
In my opinion matrix, while being open-source, federated and easy to write clients for, is still a half-locked platform because essentially the matrix.org server is "too big to fail", and the matrix team is the only one who can e.g. offer consultancy over the protocol because nobody else can have a long-term view of its future evolutions (of course the matrix people are nice, and you can discuss with them, but that is still not a standards body).
Exactly. That's why I always stress the importance of the IETF. The whole point of the IETF is that you cooperate with people, deal with opposite opinions, accept criticism, come to agreement, develop a compromise and produce a high quality absurdly documented standard. In Matrix bubble (and any others, see cryptocurrency or p2p community for example) they are the only one who makes the decision. They don't care on other opinions. If you disagree with them you're asked to stop "fighting", "focus on interaction", "understand their goals" and other meaningless stuff like "let the best standard wins" (when in reality the whole point of the standard is to cooperate). Formally speaking they try to substitute a "standard document" with an "open document" and people kinda swallow this because they don't see the difference.
> In Matrix bubble (and any others, see cryptocurrency or p2p community for example) they are the only one who makes the decision. They don't care on other opinions.
You can see our governance model at https://github.com/matrix-org/matrix-doc/blob/matthew/msc177... and see that we do everything we can to ensure that the wider community can contribute to the spec and steer it in the right direction.
And fwiw, we'd be perfectly happy to contribute the spec to IETF or W3C or whoever once it's relatively stable, just like XMPP did.
Rather than burning all this energy spreading FUD about Matrix, perhaps it might be better spent improving XMPP? :/
Yeah, didn't I say above about such kind of arguments?
I'm not a lawyer to read such documents. Just "contribute" your core to the IETF, prove me wrong.
When they don't, you get OAUTH2.
https://hueniverse.com/oauth-2-0-and-the-road-to-hell-8eec45...
Basically, chatting is message oriented, but the XMPP inventors thought it would be neat to start the connection with a tag like <beginconversation> which would then be dangling for the whole connection, preventing the use of a simple DOM XML parser (remember this is many years ago) for each message.
Just an example, here is the XML parser that I wrote in PHP https://github.com/movim/movim/blob/master/lib/moxl/src/Moxl... (127 lines), it basically handle the stream as an input and fire SimpleXML (can also be DOM XML) on the other hand.
This can handle thousands of XMPP messages (called stanza) per second :)
curl 'https://matrix.org/_matrix/client/r0/rooms/!RXNcERlITUVszBNIPW%3Amatrix.org/send/m.room.message/123?access_token='$TOKEN -X PUT --data '{"msgtype":"m.text","body":"hello world"}'
curl 'https://matrix.org/_matrix/client/r0/sync?access_token='$TOKEN
<message to="user@server.com"><body>Hello world</body></message>
HTPP's fine, I guess. It's certainly available everywhere, and saves one from writing a bunch of tiresome parsing code. But that's really an indictment of the current state of our technology. Popular languages should make it easy to read structured data out of a socket (with some sort of max length to prevent DDoSes) & write structured data to a socket. Since they don't, HTTP is an okay compromise. It's a bit heavyweight, but it gets the job done.
That said, I have quite a biased view on the topic, as I kinda blame the Matrix devs for splitting the federated communication community. XMPP is quite capable of delivering what everyone expects it to do. But lately, developers seem to be more interested in the new Matrix protocol (just because it is new). At the same time, many XMPP clients do not implement all of the latest features the standard has to offer and I wonder what the world would look like if the Matrix devs would have spent their energy on improving existing XMPP clients instead of inventing another protocol.
Edit: For completeness, you might like to take a look at the Matrix FAQ https://matrix.org/docs/guides/faq.html#what-is-the-differen...
Matrix is primarily about dealing with the issues of conferences with lots of users... I see it as more of an improved IRC than a replacement for XMPP.
How does this work? I assume a home server doesn't have a public DNS name, so what is used as the subject of the certificate then?
To issue a certificate for a specific subdomain, you just need to have an http server and serve the secret under the .well-known directory.
It's enough to drive one paranoid.
However, I would like to get my family to use something else than WhatsApp (which I refuse to use). I'm planning to do so, once I can sort out the following:
- Riot is nice if you have a lot of high-density rooms (Fractal's "Banquet" use-case [1]), but not really there in terms of UX for small rooms with my family members
- I would like to self-host if possible. However, I need to ensure that the server is reliable enough, and synapse seems to be quite resource-hungry.
Regarding the first issue, it seems that France's Tchap would solve it nicely, from what we were shown at FOSDEM. Unfortunately, the Readme in their repo is still Riot's, and it would seem that I have to build it myself [2]. I'll look a bit more into that nevertheless, and it should improve by June, from the article.
Regarding the second one, someone recently linked to this Gnome mailing list archive [3] where (in the reply) Matthew details a bit more his (then) long-terms plans for homeservers. Having one homeserver per device would solve most of my issues pretty neatly. I am not sure how far it is in the roadmap. I am also curious to know what's the status on that specific discussion topic: I was discussing just this the otherday on #postmarketos-offtopic:matrix.org [4]. I was interested in having a unique matrix daemon on a phone operating system to provide push support, maybe as well as a Matrix protocol broker.
Otherwise, for self-hosting with ~10-200 users, it looks like synapse has shed a lot of weight recently, and other homeservers implementations seem to be picking up steam, so we'll see how that goes. I am not sure how synapse would handle a dynamic IP
For now, my gripes with the ecosystem are the following:
- There doesn't seem to be easy ways of migrating: accounts, homeservers (including address and implementation, ex. migrating to/from synapse, Dendrite, Ruma...).
- No easy way to manage multiple identities: work, home, etc. Multi-account clients or community-bound identities could probably help solve this. Ideally, it would be the later, and my account would be on multiple homeservers (backup plan for availability).
- The search functionality is really lacking on most clients. No way to search my own messages, or at a specific time, in a specific community, etc.
It's also kind of fun to have both COI (chat over IMAP) and Matrix making the front page at the same time :)
[0]: https://fosdem.org/2019/schedule/event/matrix_french_state/
[1]: https://blogs.gnome.org/tbernard/2018/05/16/banquets-and-bar...
[2]: https://github.com/dinsic-pim/tchap-android
[3]: https://mail.gnome.org/archives/desktop-devel-list/2017-Sept...
[4]: https://matrix.to/#/!VTQfOrQIBniIdCuMOq:matrix.org/$15496441...
A bit off-topic, but my registrar doesn't provide an API to update the DNS records on their servers. As a workaround, would there be a free dynamic DNS with an API out there? I don't care what the address would look like, as I would just use it as a CNAME record.
On the riot side, we're trying to do our best to make it support both Barbecue as well as Banquet use cases (assuming that's not a contradiction in terms).
In terms of other points:
> Having one homeserver per device would solve most of my issues pretty neatly. I am not sure how far it is in the roadmap.
It's in the 'further still' column - we published the roadmap last week at https://matrix.org/blog/2019/02/15/publishing-the-backend-ro... - look for the card labelled 'p2p homeservers'.
> - There doesn't seem to be easy ways of migrating: accounts, homeservers (including address and implementation, ex. migrating to/from synapse, Dendrite, Ruma...).
There are scripts to do it, but it's a bit crap (similar to migrating between IMAP servers). We're working on fixing it properly via MSC1228 (https://github.com/matrix-org/matrix-doc/pull/1228)
> No easy way to manage multiple identities
Multiaccounts are next up for Riot.
> The search functionality is really lacking on most clients
yeah, we need to fix this.
What was the drive to use 1, self-host and non-federate an instance of Matrix over another platform, and 2, why? With family especially, I would assume a simple WhatsApp/Signal group would have been easier for adoption.
This is how the internet is supposed to be.
I'm not saying you personally have to build a mobile app. I am saying, in the grand scheme of things, if this is genuinely something that matters to a bunch of people, any one of them can build a mobile app and solve your problem.
There are tons of people on HN that hate Electron. If this is actually a real problem, then someone will eventually solve it, and it'll be fine. On the other hand, if a large, sizable community of native developers with tons of experience on the platforms they want 1st-class solutions for don't ever get around to solving it... then it's probably not a real problem.
I had to use their closed-source, bloated, binary blob GUI clients, couldn't logs the channels I was in (and so relied on having to log in to their servers to get backlogs, which was super annoying), and needed invites to get in to channels.
Apart from the superior multimedia capabilities of these clients, I saw absolutely no advantage of them over IRC clients (which could do useful things like logs all mentioned URLs and mentions of my nick to separate, logged tabs, and which I could run from the terminal and even from within Emacs).
I really hope Matrix is better in this regard, and I'm trying to keep an open mind, but so far attempts to create next generation chat services have been really underwhelming.
Riot is also good, but probably too bloated for your tastes :)
I've been using Riot for a few years and it's nice, but it would've been nice to know that there is a modern Weechat plugin implementation (I was half considering writing one myself).
It's also super neat to see that they wrote their own Python Matrix client library[3], which is something I also needed for another idea I'm working on (the library linked to from the Matrix site also seems to be less suitable than this one).
[1]: https://matrix.org/docs/projects/try-matrix-now [2]: https://github.com/torhve/weechat-matrix-protocol-script [3]: https://github.com/poljar/matrix-nio
The feature blurbs and API specs make it sound relevant to a project I'll be working on, but I'm concerned that there's only one viable-ish server implementation. I'd feel a lot better if there were several valid working servers in various languages by now.
For anyone interested, a detailed list of what is blocking or slowing progress can be found here: https://github.com/ruma/ruma/issues/189
Up until very recently we'd been postponing work due to immaturity of both Rust and the Matrix spec. Now that all the Matrix specs have had stable releases, the main blocker is the maturity of async/await and support for it in the library ecosystem. Since Ruma uses nightly Rust currently, we don't need async/await to be stabilized to proceed, but it does need to be far enough along that it has been adopted by all the major components of the Rust HTTP ecosystem. I'm guessing we've got about six months left until this happens.
Putting my "tech lead evaluating tools" hat on, I like the idea of a single-file binary that can be easily deployed. That said, seeing that the last commit was 2 years ago isn't encouraging :( Given that I'll likely be seriously evaluating potential approaches for our system design within the next 2-3 months, it doesn't look like Ruma would be a viable option for our situation.
There are several competing Python libraries, those you mentioned plus https://github.com/tulir/mautrix-python. We'll do a round-up and review of the options soon.
Matrix on the other hand is like having everything half-baked. This made it an non-option for daily use and I ended up moving to other IMs and a bouncer for IRC.
Also, deploying a bridge you can connect Matrix and IRC rooms, creating a hybrid channel mixing Matrix and IRC clients in the same room. It's pretty cool.