How we hosted FOSDEM 2021 on Matrix | Hacker News Reader

How we hosted FOSDEM 2021 on Matrix

matrix.org

239 pointsArathorn65 comments

Open articleView on HN

The Matrix P2P talk at FOSDEM 2021 was really interesting.

https://fosdem.org/2021/schedule/event/matrix_pinecones/

https://www.youtube.com/watch?v=a7rHtQwpuus

Unfortunately, their P2P demo doesn't work insofar as one can start a chat with you using your username or join link.

https://p2p.riot.im

Yeaah, I think it's an old bit-rotted demo using their first p2p implementation.

Still though, the progress they're making is great, and I think portable identities will be a huge deal.

The demo is still working for me on Chrome 88 (not incognito) seemingly. Which browser are you using?

It works but you can't chat with anyone. I've tried both Vivaldi and Brave. Try and start a chat with your p2p user from a hosted instance. It will fail.

This is by design. The p2p.riot.im demo is a closed system and isn't connected to the rest of the Matrix federation. It speaks a different protocol.

But that's not clear at all in the blog post announcing it or inside the demo.

Matrix is not only an exciting ecosystem to build on top off, the overall community is pretty amazing too. Fossdem went great and I liked a lot of the talks.

I'm beginning to build a couple of projects on matrix and am really hopeful for its long-term success.

Same I think the Matrix community is amazing and very welcoming. Element is really trying to build a community behind Matrix and I think they doing it with a great success. They regularly invite community members to talk in video interview (This Week in Matrix). Alternative clients are actively encouraged and promoted and small business build around the Matrix ecosystem too.

Shameless plug to the Matrix client, I'm contributing too: https://invent.kde.org/network/neochat

Unfortunately I'm running a very minimal arch + i3 setup now, I wonder if I can install Neochat without pulling in a whole lot of KDE dependencies?

Neochat looks great by the way!

Nheko is quite good too, I mostly use gomuks and nheko, though I regularly fire up neochat too :)

Not sure which are packaged, though. You usually want the latest goodies, so AUR/flatpak it is for me.

NeoChat only needs Kirigami (for the UI), KCoreAddons (for the about screen), KNotifications/KWindowSystem (for the notifications), KConfig (for the configuration). Probably the KNotifications dependency could be made optional without too much ifdef but I'm not sure if this is something users want.

If you use Weechat for IRC, there is a Matrix bridge plugin that would allow you to connect to Matrix from the command line.

Apologies if this is offtopic, but why matrix over discord? I'm just starting to learn about both (still salty about IRC's demise).

Can't comment much on feature parity beyond the fact that Matrix has had most things I need, but Discord is proprietary software and is pretty hostile towards any third-party clients. Wouldn't be a good fit for FOSDEM, imo.

Well, Discord is proprietary.

I'm not super interested in the chat/IM aspects of Matrix. What I really like is the protocol and the built-in federation and pub/sub mechanism that you can use to build cool decentralized apps that pass events around.

Discord is a for-profit unencrypted proprietary centralised walled garden. It’s forbidden to connect unofficial clients to their server, and there’s nothing stopping them from monetising your data/metadata in future.

Matrix is a non-profit open source project that defines a decentralised communication protocol that forms an open global end-to-end encrypted communication network. Anyone can run a server to participate, and anyone can use whatever client they like. You could see Matrix as descended from IRC, and Discord from AOL chatrooms.

The only advantage to Discord is that historically it has slicker UX than flagship Matrix clients like Element, more users, better room management, and exposes voice/video chat as dedicated chatrooms. Matrix is steadily catching up however.

And they can run any server they like too, of which there are many up and coming options - synapse (official / reference python implementation), dendrite (go), conduit (rust), construct (C++).

If you're an IRC fan then you might also be interested to know that the matrix team has made a pretty good effort to ensure that matrix and irc can play nice together, vs discord being hostile to any sort of third party apps as mentioned, so if your main criteria is wanting to extend longevity of irc, matrix is certainly the better option.

I attended FOSDEM this year, and the platform worked quite well. Sure, there were the occasional technical hiccups, but bugs can always be fixed if the foundations are solid, and they appeared to be. Since FOSDEM's physical environment stopped scaling years ago, I hope this will make remote or mixed remote/physical participation increasingly viable.

> mixed remote/physical

I want this for all conferences :D

Feel free to pose any questions, thoughts or feedback on how FOSDEM went, and how we might be able to do better in future if the need arises!

I was curious, how much computing infrastructure did this required? How many machines were needed (max autoscale peak)? How much bandwitdh did you consume at peak ?

We overprovisioned the hardware a lot as we really didn't want to hit any hardware limits. In practice, I think this meant there were 16 medium sized EC2 instances for the various Synapse processes; RDS for DB; 6x largish boxes in Upcloud for Jitsis, and ~70 machines in the automatic scaling group for the Jibris. I'm guessing peak bandwidth was about 100Mbps but obviously all the bulk HTTP (HLS etc) was behind CDNs.

Do you believe such a model is viable for any future conferences ? Does it need a lot of money and/or system administrators to run compared to the number of participants ?

It's definitely viable. I'm waiting to see what the bill was for the hardware costs, but I'm expecting it'll be a few $K. We had a lot of sysadmin on hand to try to keep it happy over the weekend though (but having learnt from the misadventures, subsequent events should be much easier).

Nice! Can you share about how much did it cost to host the conference then, even approximately?

I wonder if your stack can become a pattern to host large-ish to large conferences on-line!

I attended FOSDEM'21 as a speaker and the overall experience was very pleasant & smooth. No technical issues that occurred during the talk or q/a. Very well done to everyone involved!

My main issue was that the widget stopped playing when switching conversations so I had to break out the videos I listened to.

Yeah, we hooked up PIP for broadcasters but not viewers - given it's not obvious when you're switching between rooms whether you want to switch video or not. Instead, you have pop-out button to let you view the video widget separately if you want, which sounds like what you were using. The UI could be clearer.

I have no idea if that's FOSDEM infra, element.im or my home server (which is a fairly standard synapse install provided by a local tech group, so it shouldn't be out of the ordinary) to blame, but starting Saturday by joining a bunch of the relevant channels and getting told multiple times "Too many Requests!" and having to spread out joining 10-12 channels over 20 minutes wasn't fun and surprising, especially since the rooms weren't even set to provide backlog!

I generally use Matrix quite a bit, but I didn't find FOSDEM a convincing example. I guess some people like having everything squeezed into one tab, but it didn't work well for me, other than that it didn't have much of a value proposition over any other chat system for me personally, and as initially described even the experience as a chat system wasn't great.

You ran in to rate limiting, which can be configured in synapse.

All the rooms had history visibility enabled, so it sounds like something might have been going wrong there on your homeserver.

In terms of membership changes being ratelimited - yes, this is an anti-abuse mechanism, and the default synapse config is overly conservative. We don't have a way of overriding that globally though, and the FOSDEM use case of loads of rooms flying around meant some people fell over it a bunch. Sorry you got bitten by this.

Regarding the history, seems that maybe it just was very slow to sync - I can now indeed scroll back past when I joined. shrug

I didn't expect anything and was pleasantly surprised to be able to follow all talks if I wanted to, all with the room for questions and such. Congratulations, and thank you very much !

There were a few pain points, given the special nature of the rooms. Here's my workflow for joining a room:

- I have my own homeserver and am using Element Desktop

- When I'm interested in a talk, for a room I'm not already in, I need to get the room. I can't simply copy-paste it, because there's a link to https://matrix.to and I don't want to fill my own homeserver every time. I need to click on that first link _then only_ I can extract the "#room:matrix.org" I'm interested in. I wish there was just a "matrix://" or something I could click and that my system could understand.

- I joined many rooms just for FOSDEM and I didn't want to "pollute" my usual rooms. So I used the Communities thing from Element but it's counter-intuitive: in all other applications (Discord, Slack, I assume others) that button is a "namespace" at the top of a usual hierarchy (namespace/channel). In Element it's more like a tag: when I put the room in the community, I'm actually saying that the community is a view that lists all rooms with the corresponding tag. I don't know yet if I prefer this way of working or not, but maybe the elements should have a different appearance than other tools

- Linked to the above, there's no "community" for the default view, which is "all of them" if I understood correctly; I have to know that I need to click after the list, below the "+"

- Adding a room to a community is not exactly straightforward as well. I need to go in the Community, then add the room... but it must already be added beforehand, because #room:server.org doesn't seem to work.

All in all it's a lot of manual manipulations for being able to use Element the way we're expected to be able to use it. Surely your backlog already has some items for the points above, so I'm going to wait patiently :)

Matrix.to should remember your homeserver, but a "matrix://" url scheme is being worked on [1]. Work on that is caught up a bit in the backlog however, but there are folks in the ecosystem pushing that along in 3rd party clients.

Communities are going away in favor of spaces (relatively?) soon. That was the topic of Matthew's talk at FOSDEM [2], and the entire experience should be vastly better.

[1] https://github.com/matrix-org/matrix-doc/pull/2312 [2] https://www.youtube.com/watch?v=TzUfS08lMek

> there's a link to https://matrix.to and I don't want to fill my own homeserver every time.

Hm, for Element Desktop matrix.to should deeplink into the app, and so prioritise your client's settings there. If that's not working, please file a bug at https://github.com/matrix-org/matrix.to/issues.

For Element Web, the theory of matrix.to is that it remembers the URL of your preferred client (and ideally your homeserver) and so fixes up the links to do the right thing. In practice we rushed this in for FOSDEM and got it so you can provide the client in the URL (e.g. https://matrix.to/#/#misc:fosdem.org?web-instance[element.io... means "join this via the element web at https://chat.fosdem.org") - but we didn't hook it up to remember the default. I've just filed this as https://github.com/matrix-org/matrix.to/issues/197. Meanwhile, specifying a default homeserver (as opposed to client) is https://github.com/matrix-org/matrix.to/issues/15.

> I used the Communities thing from Element

Communities are a mess and being replaced by Spaces. Ironically Spaces behave the same way fundamentally (they filter your room list rather than switch you between namespaces), but we hope this will feel okay if executed better.

> There's no "community" for the default view

This should be fixed with Spaces (although quite how is still up for debate - the current behaviour still makes you 'click in the background' to unselect a Space).

So, good news: spaces should solve most of this. Meanwhile i've bumped the priority on the matrix.to issue you hit :)

I understand my own fault now: I deny all cookies by default, this might be the reason I had to enter the homeserver every time. Still, I'm not super comfortable with the idea of phoning home to someone else, even though "home" doesn't know what room I'm joining.

I haven't had the chance to attend your talk about Spaces, I'm waiting for the video. Can't wait to see how Spaces change the landscape :)

Agreed that matrix.to is a hack - hopefully the matrix:// URI scheme will get off the ground shortly and render it unnecessary. Meanwhile, the Spaces video is up at https://www.youtube.com/watch?v=TzUfS08lMek :)

> Ironically Spaces behave the same way fundamentally (they filter your room list rather than switch you between namespaces), but we hope this will feel okay if executed better.

I feel this is strictly superior in the same way tags are superior to categories. The latter imposes a strict and rigid hierarchy which quickly enters into ontological problems because you have to decide which single category a room belongs to, so your choice of categories matters too much.

On the other hand, I cannot think of a single reason why you would specifically need categories since they can be recovered by simply ensuring that each of your rooms is contained in only a single space (i.e. tagged by only a single tag). If this use case is important to a user, they can easily achieve this.

Indeed it is more powerful, as Gmail has shown us when it entered that tags are a superset of folders. Maybe I have a contact that should be both in a 'family' group and in a 'work' group.

That said using tags is also more difficult than folders, because they're a "relatively" new concept. We tech people are completely used to it, but I'm pretty sure if I ask people around me they would not be comfortable using it. The visual cues need to be different from the standard hierarchy organization.

I've seen that FOSDEM chat are archived, does element web interface allow user to save their chat room to text files on their computer ?

https://github.com/vector-im/element-web/issues/2630

This issue has 149 thumbs up, as I noted I don't know of any other chat software without the ability to save discussion to a local file.

Thanks!

Matrix itself is a conversation store, which means we haven't prioritised exporting to text file (or HTML file or whatever) in Element. A workaround for now is https://github.com/russelldavies/matrix-archive, which works really well.

Thanks for the pointer.

I installed a debian 11 VM, installed then launched the matrix-archive.py script --all-rooms with my not-for-profit ISP matrix+element credentials. I follow mostly my not-for-profit ISP channels (quite small) and may be 4 or 5 rooms on matrix.org, fosdem.org and mozilla.org.

After 30 minutes it has written 878 MBytes of archives and it's not done yet.

I don't see an option to resume / append to an existing archive. So I assume I will have to restart from scratch each day?

I wonder: will it scale if everyone uses this script?

So yes, that tool is intended for one-off archives. And I'm not surprised that the archives are large, given Matrix rooms live forever by default, so if you're in some room like #matrix:matrix.org the full history goes back to Aug 2014 will be millions of messages by now.

The 'right' solution is presumably to give Element a button to maintain ongoing archives on desktop (it already archives your encrypted rooms into an encrypted sqlite DB for indexing) - with a control to limit how far back in time it goes. Separately, having a "save the last N days/msgs of this room" button would be useful in Element Web. But both simply haven't got to the top of the todo list yet.

Hopefully I checked and I'm not subscribed to matrix:matrix.org via my local ISP matrix+element :).

I noticed the archive json files have timestamps, so may be 'archive since previous archive timestamp' would do the trick (if there's a way in the matrix protocol to do that, and if matrix software is efficient in ressources in this timestamp to timestamp case).

And having a starting timestamp being channel join time minus a user specified number of days to go backward a bit in your archive.

Since the presentation part of the videos were prerecorded, why were they not imminently uploaded to the archive after the conference?

The first videos are popping up on https://video.fosdem.org/2021 as we speak.

P.S. The FOSDEM 2021 video team was three people. With day jobs. Humans. One of us with kids. Not perfect. Volunteering our time. During a pandemic. One of us of was at a funeral service on Saturday morning. A close friend had died of covid-19 . Another had professional obligations on Sunday afternoon. It was great fun though, so chances are you'll see us again at FOSDEM 2022...

Sorry, didn't mean to imply that you should have worked harder or faster, mostly just wondered why not just release the presentation part first since that was already made in advance and the Q/A later once the editing was done.

Hey, no worries! Rereading my own comment, I realise I probably made you feel bad. That was not at all my goal. There is certainly room for improvement releasing videos early for future editions by working smarter. We'll certainly look into that!

Thanks for sharing the setup! The process seems involved enough for non-experts (even with open source tools), and clear enough for y’all who’ve done it, that I wonder whether it might be worth productizing by Element — something to the effect of a turnkey solution for conference organizers (who are domain experts but would have a hard time figuring out and managing this infrastructure setup). Given the current pandemic situation, I wouldn’t be surprised if organizers across the board jump on to a solution of this kind. It would also serve to popularize Matrix and greatly expand its user base.

Yeah, we're thinking about it. The problem is that a successful conference is much more than just the tech stack - helping guide the organisers through how to get the most out of it and then being around to support the event is a lot of the effort. My hunch is that it might be time better spent focusing on making the core of Matrix & Element better.

I did not have much time so I attended only a few talks. However, I was very pleasantly surprised how well it worked. Not only technically - it just works great having the video stream in a chat window where people can freely talk to the presenter. Maybe I'm even going to miss that on real-life FOSDEM which was inherently less interactive.

Congratulations to the Matrix and FOSDEM teams.

How much staff was required to make the behind the scenes event run smoothly? (Support, operations, helping speakers, helping attendees?)

I wonder if there has ever been an online conference better than this one. It was amazing! I really hope there will be an online-opt-in for next year

> Jibri is essentially a Chromium which views the Jitsi webapp, running in a headless X server whose framebuffer and ALSA audio is hooked up to an ffmpeg process which livestreams to the appropriate destination

So if I’m understanding this right, you have “private video conference -> web browser -> desktop capture -> ffmpeg -> public video stream"… why not “private conference -> ffmpeg -> public stream”? Having a web browser in the middle of your livestream setup feels very weird :P

The reason is that the Jitsi Meet conferencing UI exists only as a webapp. In other words, you have to render the "private conference" somehow - which means compositing together the various WebRTC streams, switching to the active talker, highlighting them, showing their names etc, rendering screenshares etc. Jitsi is an SFU (Stream Forwarding Unit), which means that the compositing is done clientside in the webapp with a bunch of HTML+JS+CSS and WebRTC video streams - so if you want to livestream a conference, you need to have a client somewhere to do that rendering. Which means needing headless chromium + ffmpeg to capture it.

The alternative would be to use a system where the conference is rendered serverside in the first place - a so called MCU (Multipoint Conferencing Unit) like FreeSWITCH. But this has a different set of tradeoffs, and having integrated both with Matrix we currently default to Jitsi.

Totally agreed that it feels unintuitive to spin up a farm of 50 headless chromiums in order to livestream a conference though!

It's not uncommon to do some webrtc video compositing (MCU) in a headless chrome, since it's usually the easiest solution and you can add some decorations with html on top. But it's definitely not very efficient. Especially since this is mostly ran on linux and chrome has almost no hardware video decode/encode capabilities on linux (starting to change).

Chaos Computer Congress on Matrix next?

If CCC were up for it, we'd be game. Could even go and embed Work Adventure into Matrix as a widget, for best of both worlds (hah). https://apfelkraut.org/2021/02/virtual-conference-experience... was an interesting side-by-side comparison of rC3 and FOSDEM.

+1 for Work Adventure !

I attended both conferences and both were great.

The traditional tool for digital communication at CCC congresses is IRC. I personally still like IRC, but I think the matrix web client at FOSDEM made channel discovery and navigation a lot easier.

I also think promoting Matrix could help with promoting open ecosystems in general. IRC works well, but it doesn't have the potential for mass adoption, which we need in order to curb WhatsApp's influence.

IRC is easy to bridge to Matrix, so that could be fairly compatible with the more conservative guys at CCC(C) ;)

The public matrix rooms at FOSDEM were bridged to irc. This was not by accident.

We tried to give visitors the freedom to use their preferred tools to join FOSDEM.

Whether you were using matrix with the latest Chromium or irssi in a tmux session on OpenBSD, you were able to speak to people at the conference.

Same for the streaming. Inside the matrix channel would have worked, but mpv on some framebuffer without X or wayland too...

Cool to see Jitsi being a key piece of software for this. Hadn't seen it used in a conference-style broadcast context before, so that's awesome!

Reply on news.ycombinator.com