Show HN: Hummingbard – decentralized communities built on Matrix
hummingbard.com
hummingbard.com
The implementation currently freestyles a bit on the event format for the messages, which is somewhat our fault on the Matrix team for running horrifically late on MSC1767 (https://github.com/matrix-org/matrix-doc/blob/matthew/msc176...) which proposes how to do rich custom event formats (e.g. blog posts) which are still renderable and visible to more generic Matrix clients (e.g. Element). As a result, if you join a Hummingbard conversation via another Matrix client currently you can't see all the messages (or vice versa). However am hoping that Hummingbard might be a trailblazer for actually exercising MSC1767 in the wild and knocking it into shape, which would hugely benefit the whole Matrix ecosystem :)
In terms of licenses, it depend on how you want the project to be used. If you want folks to run it for personal use as a decentralised social network, I'd personally go GPLv3 or AGPLv3. If you want commercial folks to be able to build on it too, then Apache is probably the best bet. copyleft + proprietary dual licensing is another option, but means you'd need any contributors to agree to the dual licensing (could be tricky), and could be seen as not being in the spirit of the GPL. that said, IANAL, YMMV.
(At the risk of going somewhat off topic, another exciting Matrix client also appeared today: https://github.com/pixlwave/Watch-The-Matrix - the first native Matrix client for Apple Watch. https://youtu.be/Jh9iewrBdGQ)
Specifically about hummingbard and/or MSC1767, have you considered borrowing/extending ActivityStreams 2.0 vocabulary? It's based on JSON-LD (proper schemas make it easier to test implementation) and could be extended to support matrix Actors/Services.
There's already a bunch of really cool stuff being developed on ActivityPub (lemmy/funkwhale/peertube/mobilizon) and Jabber/XMPP (movim/libervia). It would be great to achieve interoperability between all those.
I'll hazard a guess: New Vector LLC sees it is in their financial interest that they own the only functional implementations of the Matrix protocol and what it can interact with, so there's no interest in federation with the wider Fediverse and other protocols.
Now Arathorn, come tell me why I'm wrong
The reason I skipped over this was because the author seemed to be pushing an agenda elsewhere in the thread ("Matrix is like XMPP but with a heavier server": https://news.ycombinator.com/item?id=26279816, and "Matrix has one reference client/server with really good features but hardware requirements that place it out of ordinary folks' reach.": https://news.ycombinator.com/item?id=26279780) which made me wonder whether answering all their posts was going to be a good use of time. In practice, the thread hasn't exploded into a XMPP v. Matrix holy war, so it looks like my concerns were ill-founded.
To actually answer the question from the post in question:
> Is there a working group talking about vocabulary or interop with other federated networks (ActivityPub/XMPP)?
No, there isn't a formal WG - instead there are a set of informal conversations between the 3 projects. I lurk on xsf@muc.xmpp.org and standards@xmpp.org in case interop discussions come up; I also hang around various cross-project rooms where Gargron and Christopher Allen Lemmer and others are active on the AP side. As a result, cross-project initiatives sometimes pop up - for instance Matrix supported Mastodon in adding E2EE at https://github.com/tootsuite/mastodon/pull/13820, and we've also tried to work with XMPP on using Olm for E2EE as an alternative to libsignalprotocol in OMEMO.
In terms of content vocabulary specifically: we researched JSON-LD a bunch when working on MSC1767 (back when it was MSC1225) - the notes predate us using GFM for MSCs and are archived at https://docs.google.com/document/d/1m4VTRqclB3JEMZBjbr4t5cvI...
In practice, I'm not convinced there's a huge amount of value of standardising on the same vocabulary though - whether that's MIME, JSON-LD, RDF-flavoured JSON-LD, IPLD, rich XMPP stanzas, etc - as long as there's a way to map between them. Given you need to map the protocol layer in general when bridging protocols, why not map the content format too? That said, if folks have solved the problem elsewhere satisfactorily, then we'd have no problem with piggybacking on it on Matrix (just as others have piggybacked on Matrix's E2EE).
In terms of the parent's comment:
> I'll hazard a guess: New Vector LLC sees it is in their financial interest that they own the only functional implementations of the Matrix protocol and what it can interact with, so there's no interest in federation with the wider Fediverse and other protocols.
No, my priority is trying to make the wider Matrix ecosystem as diverse and healthy as possible, including bridging nicely other protocols into Matrix. The more different projects building on Matrix (e.g. hummingbard) the more potential in the network and protocol for everyone involved. A rising tide lifts all ships, as it were. My dream for Matrix is to become the realtime communication layer of the open Web, and for there to be as many different parties building on it as do the open Web today. If we pull that off, then Element (formerly known as New Vector) would hopefully benefit from it as much as everyone else - much as 3Com benefited from creating Ethernet and bootstrapping that industry alongside DEC, Intel, Xerox and others.
But it would be UTTERLY STUPID for Element to privilege itself over other contributors to Matrix, and would effectively self-sabotage the open ecosystem, thus killing the main thing which could make Element successful in the long term. Hence the huge amount of effort and $ that we put into setting up The Matrix.org Foundation (https://matrix.org/foundation) as a neutral independent guardian of the protocol, and protect it from anyone commercially building on Matrix... including Element/New Vector.
Sorry you think we suck though :)
I have some grievances with Matrix. I've aired them in other threads, and on the Matrix.org server. Some have been addressed, others have not. I really hope I was constructive in my criticism.
Frankly, some members of the community here act like absolute children every time Matrix is discussed, and hold it to impossibly high standards (like complaining it isn't as usable or polished as privately owned VC competitors, or older standards that have been around for 5x as long and can't do half of what Matrix can). It's disappointing and frustrating because I'm afraid that you will be less willing to engage here, and upset that it detracts from actual critical conversation of the protocol.
Anyways, thank you for what you do and continuing to engage here.
Separately, it's true that Matrix intrinsically uses more resources than XMPP, because Matrix's architecture is one of replicating and persisting conversation history and state between servers, rather than passing messages. And this simply involves much more data and processing than just relaying messages between sets of users.
Thanks for replying, that is definitely not my intention. My main concerns are interop, privacy and usability. I have technical/political criticism on both sides (XMPP/matrix) but as an end-user/sysadmin i don't see much difference in spirit/protocol between XMPP and matrix (except for decentralized rooms).
I'm currently using Jabber/XMPP for technical reasons. My two blocking issues with matrix are client side with element being really latency-sensitive (+ > 9MB loading), and server-side with synapse taking resources so the non-profits i know (eg. tildeverse.org/chatons.org/libreho.st) only provide Jabber/XMPP service. But like i said elsewhere in the thread i'll give dendrite a go.
My agenda is not hidden and is as simple as empowering users to federate across decentralized networks/protocols so we can stop arguing which solution is the best and stand up together against the corporate silos. This was stated in the goals of the joinjabber.org collective we founded a few weeks back: https://joinjabber.org/collective/about/goals/
I can assure you i have bitterness on both sides, with XSF mostly concerned with business use-cases (not free/non-profit ecosystem), and with matrix investing considerable resources into reinventing the same wheel as XMPP did two decades ago (a universal federated, bridgeable messenger). Despite everything, my agenda is as simple as "interoperability".
> as long as there's a way to map between them
I agree on paper. But in practice very few projects actually do that (only Hubzilla/Bridgy to my knowledge) which led to my question of potentially reusing an existing standard instead of coming up with an entirely new vocabulary. In my personal view, coming up with new vocabulary is also a valid approach if interop is not an afterthought.
> if folks have solved the problem elsewhere satisfactorily (...) to become the realtime communication layer of the open Web
I believe ActivityPub/ActivityStreams has solved many problems, and is already the most advanced/standardized federated social protocol on the open web (it's a W3C recommendation) despite not being especially real-time oriented. There is room for interpretation/improvement in the specs but "user profiles, posts, communities, sharing" (as in hummingbard) is definitely covered by major implementations in the fediverse even if "communities" use-case was not explicitly mentioned in the initial AP spec.
Still i made the effort to follow your link to Google Docs (seriously why Google? i made an exception for you but it's fucked up) and i better understand your concerns about reusing/mapping ActivityStreams to matrix vocabulary. So while i can't say i encourage you to reinvent yet another social federation vocabulary, having a strict mapping like you mentioned in the doc makes sense... but it should really be priority n°1 otherwise we're just splitting the ecosystem even more (see XKCD about standards).
Interoperability for social networks is really cool. People enjoy being able to comment on a Peertube video directly from their Mastodon client/account. I hope we can have matrix spaces (and hummingbard and others) part of this federated future as well. I don't believe standardizing a gateway/bridging mechanism is enough if interoperability is an afterthought (in the rest of the standards) because that's precisely the path that Jabber/XMPP took ~15y ago and it's failed dramatically.
Take care and keep in mind despite our respective opinions/criticisms, we all care for empowering and protecting users. Our enemies are corporations, governments and other kinds of oppressors, not other federated protocols.
what will you see if you open it in Element?
Ie is it a series of messages? Or is it something more advanced; with built in conflict resolution or other P2P-y features?
Regular Matrix is pretty straight-forward - you join rooms and send events. Events get propagated to all participating/federated servers that are in the room. You watch for incoming events.
The Matrix devs should be able to say more about their P2P implementation. :)
I can't tell what language Hummingbard is coded in. Hint?
GPL with targeted relaxations is usually a good way to start. You can always relax the license more later if you exercise the foresight to get contributors to allow relaxations per your judgment; and, don't rely on any strict GPL dependencies. Being GPL but not able to use GPL libraries is a funny position to get into, but that's our world.
If you want forks accessed via the network to publish source code, use the AGPL, otherwise you may as well just use the MIT licence.
From the client's perspective, matrix is just a basic stream of events (like XMPP). The magic happens on the server side where a room is replicated across servers and conflict-resolution voodoo is applied. That's also why synapse server is really really resource-intensive (gigabytes of RAM) compared to equivalent-in-features (apart from decentralized rooms) XMPP/ActivityPub servers ; dendrite is fairing better but i haven't benchmarked it yet.
Synapse’s RAM usage is a) way better these days, b) not due primarily to the merge resolution algorithm (state res), but due to slapping in-memory caches everywhere rather than being smarter on the db schema. Dendrite does 5x better by being smarter in the schema, and Conduit looks to be even better via cunning use of using key/value dbs for persistence (sled).
Eg, for sake of simplicity, imagine you're storing Git in Matrix (similar to what i'm doing, if i use Matrix). I've got two primary concepts:
1. A mutable set of pointers.
2. A series of key values, where the keys are content addresses, and the values are a byte array.
My initial assumption would be to use the messages to post all newly created blocks and changes of pointers. Eg, Send a message with a new content address and the bytes. Send a message with a new address for a pointer.
For a Git-like to work in that model, it just needs to slurp up any messages in the datastream. Making sure to also slurp in old messages properly that maybe are out of order (ie, reconnecting after being offline for a while). But being a Git-like, it's a fairly simple model.
With that said, perhaps the metadata storage in Matrix would better suite some of this? I just imagine Matrix isn't intended to host TiBs of `/state` metadata.
Thoughts?
In the longer term we'd (well, I'd) love to make it work well with freeform KV data (or object graphs! you gotta store your VR scene graphs in Matrix, after all) and act like some kind of crazy global zero-trust CouchDB or Cassandra. But right now the KV state is meant to be for relatively simple freeform metadata about a room.
That would be great for a decentralized forge based on matrix. This is an area where i think matrix decentralized rooms would shine, as they would effectively prevent DMCA takedowns for software projects.
Thanks for the clarification. My point was that building something client-side for matrix is pretty straightforward, in case that was not clear.
> not due primarily to the merge resolution algorithm (state res), but due to slapping in-memory caches
Good to know. conduit doesn't quite look to be ready, but would you recommend using dendrite in production to serve a few users? i guess i'll give it a go
> Hummingbard’s code will be available as soon as I am able to decide which license works best for us.
Generally I think MIT is a good default and you should only use a different license if you have a strong case for why MIT doesn't make sense for your project.
I think you have a real product here and would caution you against releasing the source code.
If you do want to open source it, I highly recommend GPLv2 or 3.
This is solid advice for client software. However for software to run on the server side, AGPL providers better protections to ensure further developments remain free (copyleft).
The choice of license really depends on what the OP aims to achieve with his software.
That is simply not true. It may be true for huge corporations with legal counsel spreading FUD. But why should we care about these people ruining our lives? They're already destroying the planet and making sure millions of people are enslaved in the mines/factories across the third world, and we should provide free work for them?!
Interesting entities (libraries, schools, associations, workers coops) will not care about the software license.
I assure you, even "interesting" entities care about this. Hell, even public institutions I worked for did.
For what it's worth, I think there's reasoning to consider it on a project-by-project basis. An example might be how XMPP more or less became a bunch of walled-garden forks for various messengers like Google Chat and Facebook. I think Linux thrived with the GPLv2's copyleft being a "sweet spot" for them.
OP, if you see your project switching licenses down the road, you might as well look into CLAs, a whole other can of worms, or just stick with the least restrictive license that achieves your goals. I'd personally use LGPLv3.
No: it's their choice, and if they choose not to use GPL software it is evidence that they do not care about user and developer software freedom. And that they don't want to contribute back.
Also, "some major institutions" is very unclear. Google is ok with GPLv2 but not with GPLv3. Apple has restrictions on their app store.
Both companies are extremely unlikely to support decentralized communities anyways - they outright compete against them.
This is still possible with GPL (and the AGPL, for that matter) by granting an exemption to the license for third-party extension software that is in the form of a plugin using a specific API.
It is worth noting that depending on the details of your (and the plugin's) implementation, such an exemption may not even be required.
This is probably a good place to start digging in to the topic: https://www.gnu.org/licenses/gpl-faq.html#GPLAndPlugins
But before anyone starts down this rabbit hole (the legal, social, and technical details of which I personally find fascinating, but YMMV), a maintainer should instead think about the things they want to allow, the things they want to forbid, and the things they want to be able to charge for. Also, which of the aforementioned things they want to provide strong guarantees for in the event of a change in ownership and/or a fork (eg. MySQL/Oracle, OpenOffice/LibreOffice, etc.).
But whatever the preferences are for a particular project, there almost certainly is a way to set those boundaries using a FLOSS license, and it is probably possible while using the GPL (but if you're leaning toward 'allow individuals and mega corps to do whatever they want', there isn't much sense in using the GPL as a starting point), though it is possible you might need a few additional tools for some scenarios (requiring a Contributor License Agreement or even copyright assignment for contributions, license exemptions for specific APIs, splitting out some shared libraries and releasing those under a different-but-compatible license, dual licensing, separate licensing for non-code assets, a nonprofit foundation, compatibility testing and associated badging & trademarks, etc.).
Anyway, there are a lot of tools available, and most of them can be grabbed when and as they're needed, if they ever are.
So, start with what you want in terms of permissions and boundaries, then ask for advice on enabling/enforcing that structure with a FLOSS license, and it can probably be done without too much complexity (assuming that it isn't incompatible with FLOSS, like 'This software cannot be used by the military or to support the creation of WMDs').
I don't know of any copy left license that prevents you from making JSON API calls from any source you want, so you can integrate something like matrix easily enough.
XMPP/Matrix went the first way. Matrix ecosystem has one reference client/server with really good features but hardware requirements that place it out of ordinary folks' reach. Similarly, XMPP has some really good apps built on top (eg. Whatsapp) but the free-software clients (not servers) are lagging behind, though catching up quickly in the past years.
ActivityPub ecosystem went the other way with AGPL everywhere (Mastodon/Epicyon/Funkwhale/Peertube/PixelFed/Funkwhale/Mobilizon/WriteFreely/Lemmy/Plume) with the notable of bookwyrm who has the Anticapitalist Software license. As a consequence of the ecosystem tailoring to users/non-profits/cooperatives not corporations/governments, it's blooming into this amazing and friendly ecosystem where everybody contributes and interoperates. Hell, who would have thought three years ago selfhosting federated/p2p livestreaming video would be become so easy as Peertube?
Personally i'm more on Jabber/XMPP's side (i contribute to joinjabber.org project, and use Jabbber/XMPP daily) mostly for technical reasons (AP/matrix haven't yet caught up with XMPP in many regards) but in my view adopting loose licensing is what almost killed the XMPP ecosystem all these years ago when it had the exact same promises as matrix does today: one universal federated protocol to bridge to all others.
Define ordinary folks, please. Matrix isn't really that resource intensive for a small homeserver between family and friends.
Server-side, many smaller hosting coops went into matrix but later dropped it because of synapse using too much resources. I know individual selfhosters who were really sad to drop it because their raspberry pi didn't have enough RAM (512MB-1GB) for selfhosting synapse.
However someone said in another thread that synapse resource usage is really better these days. Would you recommend it on such low-end hardware? (ActivityPub/XMPP servers fare really well in such conditions)
$ ldd `which element-desktop` | wc -l
102I've heard of people running Synapse on a RPi 3 (see https://old.reddit.com/r/selfhosted/comments/g9h6au/matrixri... for instance) for such use cases. If you want to chat in large 1000+ member rooms, it probably won't be enough. However, even for such a use case, you don't need anything that is out of reach of a common person, just a few gigabytes of RAM. In general, it's hard to make it go over 3 GiB. My server hovers around 2 GiB and I'm in many large rooms, including the beast that is Matrix HQ.
So I would agree that Synapse is not usable on RPi-level hardware in all use cases, but you can definitely run it on such hardware for some use cases. Dendrite is also maturing really quickly and will be less resource hungry.
It's frustratingly slow and heavy, regardless of client.
If you grep the logs for state-res you will probably see that some room is consistently chewing resources (these days we explicitly log the worst offenders); easiest bet is to ask your users to leave that room or use the shutdown api to remove it from the server.
Otherwise, it may be that there’s so much data flying around due to the busy rooms that the in-memory caches are blowing out. This makes everything slow down as the DB gets hammered, and unintuitively uses more RAM as slow requests stack up. The solution is to increase the cache factor, much as you would on an overloaded db. We’re currently looking at autotuning caches do you don’t have to do this manually.
If it’s still slow, then there’s probably a bug or other weirdness (ancient Python?) - my personal server has about 5 users on a 4 core cpu and uses about 2GB of RAM without a dedicated DB node and is in thousands of rooms (including busy ones).
(Also it hopefully goes without saying that all bets are off if you aren’t on the latest Synapse release - we are constantly landing improvements currently; eg the auth chain cover algorithm eliminates most of the known perf edge cases on state resolution).
Sounds like I should tune some caches then - I have memory to spare if it turns out to make a difference.
BTW, I just noticed there is an option to add a Redis - would that be a significant improvement compared to just using the process caching?
Redis is only useful if you split the server into multiple worker processes, which you shouldn’t need to at that size (and even then, doesn’t provide shared caching yet, although there’s a PR in flight for it - we currently just use redis as a pubsub mechanism between the workers).
Highly recommend hooking up prometheus and grafana if you haven’t already, as it will likely show a smoking gun of whatever the failure mode is.
Are the logs stacking up with slow state-res warnings? Stuff like:
2021-02-25 23:15:26,408 - synapse.state.metrics - 705 - DEBUG - None - 1 biggest rooms for state-res by CPU time: ['!YynUnYHpqlHuoTAjsp:matrix.org (34.6265s)']
2021-02-25 23:15:26,411 - synapse.state.metrics - 705 - DEBUG - None - 1 biggest rooms for state-res by DB time: ['!YynUnYHpqlHuoTAjsp:matrix.org (148.6s)']Citation needed. The European Union Public License is copyleft and it's literally written by governmental bodies.
Even a government like france which is involved in strong open-washing campaign has serious Microsoft ties behind the scenes (education & military mostly) and despite all the promises for 100% OPEN, keeps on introducing closed-source algorithm which are paid for with millions of euros of public money and whose results are very sketchy to say the least (i'm looking at you ParcourSup).
So, when i said "less friendly to governments", i did not mean from a legal perspective. I meant AGPL is only one expression of a global approach to software development that is focused on user empowerment. Such software is usually less concerned with problems of big organizations (including governments), and less likely to try and reinforce governmental control (which is something most governments are looking for in a product). Does that make sense?
Definitely not: https://matrix.org/blog/2018/04/26/matrix-and-riot-confirmed...
And having worked for the last 6 months in a French governmental agency, Tchap[1] is the only internal instant messaging solution I was made aware of. Most people still had an email+phone centred work-flow however.
https://choosealicense.com/licenses/ provides a concise rundown of a few common ones
https://www.gnu.org/licenses/license-list.en.html: a more detailed list
https://github.com/plibither8/licensed: a CLI tool to help choose licenses (we're on HN).
As for the password, Hummingbard needs it for the same reason that a server-hosted webmail client like Roundcube needs your email password to log in.
For more signups, it would be great to support social logins. That has been implemented in some version of Matrix, but I’m not sure if it’s in Dendrite yet: https://github.com/vector-im/element-android/issues/2452
With an app (sometime soon, I hope), there’s a better chance of getting more users to adopt it. I’ve been waiting for Pixelfed (decentralized alternative to Instagram) to release an iOS app, but it still hasn’t seen the light of the day.
Pretty interesting. I was putting a little effort to get a Dendrite homeserver running a month or two ago and it was non-trivial at that time for me - but evidently the author feels it's stable enough to run an experimental project like this.