Ruma, a Matrix homeserver written in Rust
ruma.io
ruma.io
[1] https://github.com/ruma/ruma [2] https://github.com/ruma/ruma/blob/04e5af143136c026c1b7449921...
(Seriously, I know it seems minor, but I've always hated the name they picked; it's almost like they named it "filth".)
The documents I know of that provide details about it are here:
* https://matrix.org/jira/browse/SPEC-162 (the tracking issue for the feature in Matrix's issue tracker)
* https://matrix.org/git/olm (C implementation of the protocol formerly known as axolotl v2 developed for use in Matrix)
* https://matrix.org/speculator/spec/drafts%2Fe2e/client_serve... (draft of the spec for E2E)
Matrix is lacking its E2E encryption but already has everything else. You can use Vector today to replace Hangouts or Skype completely.
There's no concept of finding "peers" in Matrix. It's more akin to xmpp : each user is attached to a Homeserver, which can chose to federate to one or multiple other HSes. Users may then create rooms on their own HS (or the HS they federate to) and other users may join those rooms, similar to IRC.
[0] http://matrix.org/speculator/spec/drafts%2Fe2e/client_server...
Or am I wrong, and if so what did I miss?
Edit: Don't get me wrong, I'm dreaming to see the end of that fragmentation happening, and I don't want to discourage anyone to address that problem, I'm simply wondering if anything is fundamentally different from XMPP.
Edit2: found a few answers here : https://matrix.org/docs/guides/faq.html#what-is-the-differen...
Happy to follow up with additional questions, though!
I know there are ways to get around but I don't see any which is sufficiently good for me (e.g. low boilerplate, scalable/composable, no compromise of performance). Some details are in my question on StackOverflow[1].
One particular example that C and C++ allow is to have an object "self-destruct" itself in a callback (e.g. after a TCP disconnect, EventLoop callbacks Socket callbacks MyClient which directly destroys MyClient including Socket). Yes it is indeed valid to call a (virtual) function on a class and for that class to proceed to delete itself [2]. This is very nice because it removes need to have special cleanup code all around and/or half-dead states. In Rust, this seems impossible by design (i.e. would need unsafe, and it's unclear if the compiler itself allows it anyway).
[1] http://stackoverflow.com/questions/36952894/event-driven-des...
[2] http://stackoverflow.com/questions/3150942/c-delete-this
The main app is C that links all my libs together, but the actual C code just calls a Nim to execute the code. I talk to all the C libs using Nim.
I'm very happy with that decision. It was a breeze to setup and is great to work with.
i don't follow - Rust integrates with C quite well, too.
You can do what you describe as long as you use Rust's reference counting support (Rc and Arc), which includes weak references that can be used for parent pointers, plus RefCell and Mutex for mutation.
To have an object "self-destruct", have it remove all reference counted pointers to itself.
You can pass an &Rc<Self> (or &Rc<RefCell<Self>> or &Arc<Mutex<Self>>) as the self parameter if you want to let an object create references to itself. If you want an object to be able to drop itself immediately in that case, use an Rc<Self> instead of an &Rc<Self>, and call drop(self) in the method (this also work if you are not using reference counting and just pass Self, of course).
You should try to not do this kind of thing though, because it adds overhead and the compiler cannot statically check that you are not leaking reference counted cycles or deadlocking on mutexes or refcells (which is not a Rust limitation, it's just impossible without having the programmer write a machine-checkable proof).
If you do it in C++ the compiler also cannot check whether you are referencing freed memory or incorrectly concurrently modifying the same object.
struct MyFancySharedThing<T> { thing: Rc<RefCell<T>> }
Rc and RefCell are implementation details and should _always_ be hidden.
Is there a recommendation on what to do instead?
In Rust you'd have to either make an few (unsafe) global ( https://github.com/rust-lang-nursery/lazy-static.rs ) structures to keep track of the event-driven state, or pass them into every callback, or make every callback somehow derive (or a derived from) a common structure that does this bookkeeping.
So far the language doesn't have syntactic sugar for this, but I think it'll be there in a few years. The compiler is up to the task (as you can already see a few 3rd party macro-driven solutions for similar "magic" things - such as serde's custom macros).
But since every event-driven thing can be implemented as a queue and a consumer thread pool (as it's implemented under the event-driven hood), I don't think Rust is a non-starter for even-driven solutions. Though I'm inclined to agree on the extra care needed to satisfy the type system to get easy callbacks is annoying.
A few hints how to do it:
- no shared_ptr / ref counting, make object ownership clear
- design to allow destroying an object at any time even from a callback
- don't make callbacks harder than they need to be (use virtual functions or simple macro hackery to reduce boilerplate of using function pointers; don't introduce "signals and slots")
On the other hand, with function pointers or virtual functions, you can easily ensure a required callback is provided, by requiring the user to pass it in the constructor.
I don't see any difference related to "accidentally passing the wrong function". In either case you need to specify which function to call and on which object; for virtual function callbacks it's even easier and harder to fail. Type checking can also be done by the compiler in either case (even for function pointers at least in C++ it is possible to make a type-safe wrapper, see my implementation of lightweight callbacks [1]).
[1] https://github.com/ambrop72/aprinter/blob/master/aprinter/ba...
I'm coming from Python, so the odd simple integer/boolean check (what Rc and RefCell come down to) aren't an issue for me - they might be for you depending on what you're writing, I suppose, although 99% of the time they're not going to be what you need to optimise.
However - Rust is not an object-oriented language. This sort of design isn't necessarily what you want to be using. In your particular case, what you'd probably want is the EventLoop owning the Socket and MyClient, callbacking MyClient directly, and allowing MyClient's callback to return a value describing whether to destroy it or not. Libraries like rotor[0] do exactly this.
There's also a chance that you might be able to encapsulate your unsafe code behind safe abstractions, and Rust can help prove the rest is memory-safe.
Wrapping everything into Rc<Refcell<>> and lots of dereferencing was one turn down (yes - C++ also needs shared_ptr<>) from the syntactic and ergonomic point of view. That might now be better due to some automatic derefing. The biggest issue that I had was that reference counted 'interfaces' (trait objects in rust) were not working in the required way at all (no required up and downcasting of boxed trait objects was possible). Don't know how this changed since then.
If I would need to perform the task again I would probably try to model everything with synchronous/blocking API calls as this seems to fit much better into Rusts ownership model, even thought this would sometimes require lots of threads.
Specifically, I've found that is possible to prevent any callback-related use-after-free and related callback hazards by following these simple rules:
- Callbacks should always originate from the the lower layers (event loop), should never be called back synchronously as part of a call from upper layers. If you need to callback soon, use a facility of the event loop that makes the event loop call you soon.
- When invoking a callback, "return" immediately afterward. If you have the desire to do something after invoking a callback, do so by first queuing up a call to you from the event loop using the same facility as mentioned above. Actually, callback typically return void , I usually call them by "returning them", i.e. return callback();
- Make good use of destructors or related facility (if your language doesn't literally have destructors) to ensure that when an object is destroyed, it will not receive any more callbacks from the objects that it used to own.
I think that a programming language could even enforce these rules at compile time (and the logic for this is much simpler than rust's ownership system).
As I see it, its basically letting you ignore the context code is called in, but the issue with that is, well, IME failing to understand that properly is where nearly all bugs come from.
It's true that rusts rules about aliasing are frequently, well, annoying, but its hard for me to think that poor support for event driven programming is that bad of a thing.
... FWIW, you should possibly take a look at servo. Browsers have to support some level of eventing, so it seems like they probably have a system for this.
I do struggle with the ownership rules sometimes but it's pretty much always because I'm clashing with the bad habits I learned coding in C/C++, or I'm shooting my future-self in the foot because my limited cognitive capabilities can't always keep all code paths in memory perfectly all the time.
The borrow checker came with an upfront cognitive cost that ultimately saved me from multithreaded async I/O madness later on. I can sleep at night knowing my multithreaded async system is more sound and easier to safely extend/maintain than it would have been had I created it in C/C++.
I can see why some people might give up on Rust. In C++ you can take the easy way out (use mutable aliasing etc.) and you can quickly get things 'working'. You don't have it so easy in Rust because you have to carefully think about end-end ownership.
Also in Rust I found that when I did make bad design decisions it was sometimes a lot more work (redesigning structs, shifting ownership responsibility) than it would have been in C/C++ (because I probably would have laid land-mines for my future self by mutably aliasing things etc.).
EDIT: again, not trying to put anybody down, I would just like to learn if this code is representative of bigger Rust apps.
EDIT: I think what gives the code its length here is the explicit error handling. If you do the same in Python you'll end up with the same amount of code. In Rust, you could unwrap() everything and lose nice error messages and probably slash the LOC in half.
let value = try!(fn_that_may_error());
The try! macro would make the linked function much less verbose... Except that what try! does is bubble the error further up the stack by returning an Err if the called function returns an Err — and since this is the main function, there's nowhere up the stack to go (main() has a return type of "Unit," which is Rust's version of void or null; for functions that pass errors up the stack you'd have a return type of Result<ReturnType, ErrorType>, which allows errors to be returned), so try! can't work.
I agree that some of that error handling in 79-101 is pretty awkward looking, but luckily it's not super representative of idiomatic Rust: usually you can make Rust much more concise by using try!, with the exception of the main() function which can't use try! and has to actually handle the error.
The first section of code you mention is using what's referred to as the "builder pattern" in Rust, where you have a dedicated type that chains method calls to mutably construct an object which is produced at the end. You're not alone in thinking this pattern is ugly and verbose (though I've grown to like it, personally.) There is a lot of discussion[1] about providing a terser approach via keyword arguments, but nothing concrete is planned at this point.
The second section of code you mention is a match statement (essentially a fancy, type-based switch statement) that inspects the arguments presented to Ruma from the command line and dispatches to the appropriate code. This supports two subcommands currently, `run` and `secret`, which are the two arms of the match. The `run` arm loads the user's configuration from the configuration file into a Rust structure, then starts the web server, which blocks the process until it is killed, reporting any errors that occurred along the way. Rust is very explicit about code paths that are optional or may produce errors, so you often see the "some" and "none" (or "ok" and "error") cases handled explictly in code, which will certainly look verbose coming from languages like Python. I came from Ruby, myself, and what I've learned is that the places where Rust feels weighty are almost always making me accountable in places where I would've otherwise left a corner case unhandled or introduced a bug in a Ruby program. There are some constructs in the language to make handling optional and error cases a little less verbose (e.g. `unwrap_or`, `try!`, etc.) but in many cases I prefer the longer form for simplicity.
Lines 78-107 are "do stuff or handle error". You could do "do stuff or crash" instead more tersely via thing_that_might_fail.unwrap().
I think lines 80-87 for example are idiomatic. Lines 89-100 seem a little strange to me in that they embed one error case in an ok case. I would probably do one error per block, returning the server and then using it in a separate block. But I'm new to Rust so don't take my word as gospel.
In any case, I think it's true in any language that when you explicitly handle every error in a unique way, your code is longer. I suppose one way to shorten would be to make each error-handling case more similar: just print the error without a unique prefix. Then make a function for each subcommand that returns Result<(), Error> (nothing on success, error on failure). Then the functions could use the try! macro to make the error-handling implicit (the macro automatically propagates error to the caller), and main() could handle all the errors with just one path. Something like:
fn run() -> Result<(), Error> {
let config = try!(Config::from_file());
let server = try!(Server::new(&config));
server.run()
}
fn secret() -> Result<(), Error> {
let key = try!(generate_macaroon_secret_key());
println!("{}", key);
Ok(())
}
fn main() {
...
let result = match matches.subcommand() {
("run", Some(_)) => run(),
("secret", Some(_)) => secret(),
_ => Err(SomeErrorType::new("no such subcommand")),
};
if let Err(e) = result {
println!("error: {}", e);
// or to stderr via the ugly:
// writeln!(&mut stderr, "error: {}", e).unwrap();
process::exit(1);
}
}
Maybe we just need some federated system that's a directory only. All it does is help you find people and resources, after which they speak end to end.
In terms of the community size - looking at just the Matrix.org homeserver, there are around 300K messages a day, 250K users (of which about 200K are bridged), and 30K rooms. Meanwhile there are at least 500 openly federated homeserver installations we can see from the Matrix.org homeserver.
Synapse is relatively mature as a homeserver (although it still has performance challenges; it is very cool to see Ruma progressing!) - meanwhile Vector as a flagship web/ios/android client is in late public beta and very usable too - eapecially with end-to-end encryption on the horizon.
I may be biased (being project lead for Matrix), but I'd say that it's going somewhere :)
Is anyone already using Matrix as an event server/queue for interactive environments and/or sensor networks (similar to the Stanford Event Heap [1])?
What is the minimum latency for Synapse/Ruma?
However, this is still fairly PoC. Our latency is deliberately high at the moment (given all events are persisted on all participating servers, and signed etc) - typically around the 100-300ms mark depending on server performance involved. If you want lower latency stuff (eg VoIP or MIDI) the architecture is that you use Matrix to be a signalling layer to negotiate the realtime protocol (eg RTP).
Nope, Friendica did that first (started in 2010): http://friendica.com/
Right now most of the clients are geared up for the messaging and VoIP use cases.
The value prop is to provide a meta-network which connects together all the existing communication silos, so users can talk to folks on other platforms (eg Slack, IRC) without caring what service or client they are using. Ideally someone on Slack could end up talking to someone on IRC, bridged via a decentralised Matrix room, without even realising they are using Matrix. Alternatively you could use a native Matrix client like Vector.im or the Matrix API to get at the data. The key thing is that no single silo ends up owning or controlling the data - it is replicated over all the participants, who thus own the conversation equally.
The end goal is to let users select which communication apps and services they want to trust with their data (including running their own, if they so desire), rather than the communication and contacts being fragmented over hundreds of different apps.
https://roamingaroundatrandom.wordpress.com/2014/06/01/a-dec...
It isn't very technically detailed, but it should be clear enough. Your system sounds like a good match for what I imagine. The TL;DR: The messages are in focus, not the servers. Content-addressing using hashes, threads managed through messages referencing prior messages directly by their hashes. When creating forums, servers are just (optionally) defined to simplify routing / delivery.
One could build discussion servers on top of the matrix home server, and discussion clients to match.
[1] https://www.ruma.io/docs/matrix/ [2] https://www.ruma.io/docs/matrix/why/
These things do matter to businesses though. I've seen multiple companies adopting Slack express discomfort over the idea that the communications are hosted and archived by Slack. Email is broken and horrible, but businesses still use it for a lot of the same reasons they might choose Matrix. Slack fixes the horrible, but lacks (and its business model is likely incompatible with) a federated protocol. There could be an opportunity there.
That wouldn't be one company though, that'd be hundreds of different companies all providing their own clients and hosting solutions.
Perhaps a good comparison is the public reaction to Snowden's leaks about the NSA. They haven't been super big news outside of tech and certain political circles. The average person values convenience and a good user experience over any more philosophical or political beliefs related to privacy and security. The idea of a centralized service or company controlling data just doesn't matter to a lot of people. You're absolutely right that the business motivation is not quite there, because there isn't a strong public demand for it.
Matrix itself does have some financial incentives—specifically, its development is funded in part by a company called OpenMarket, who are also the developers of Vector, a web-based client for Matrix that will at some point have a commercial product offering.
Edit: In other words, Vector is planning to do exactly what you describe: provide a product that competes with Slack, but building it on a protocol where someone could create a competitor using the same protocol, allowing them to be interoperable. Ruma does not have a financial stake in the game, and I've chosen to build on Matrix because it is well aligned with my values.
Good -- if someone demonstrates that it's possible to make money with federated messaging systems, then we can see better investments in them.
Also I was just trying to implement our own Telegram Server to take advantage of Free Telegram Clients available.
Can anyone compare Telegram with Matrix. (Except the Federated part).
More detail in Matrix's FAQ: https://matrix.org/docs/guides/faq.html#what-is-the-differen...
If Wave had bridged with protocols like email or IM, provided a clearer UX for newbies, and nurtured an open community and network rather than focusing primarily on Google's use cases, Matrix probably wouldn't be needed today.
Neat project though.
This comic gets linked quite a lot when discussing new systems, and I and many others see it as a fairly low-effort jab. I'd venture a guess that anyone reading HN has seen this comic before, and the creators of systems like Matrix are certainly not oblivious to it as common criticism.
While there is danger of becoming yet another variation that doesn't "win" or solve problems in a meaningful way, the folks developing systems like Matrix think they can do more to help by approaching these problems in a new way than they can by throwing up their hands and living with the problems, and I think that effort deserves respect.
It'd be more constructive to do some reading about the protocol, become more familiar with it, and share more specific criticisms that concern you than to drop a link to the comic everyone has seen before.
It's something that a lot of people (including myself) really, really want to succeed, but all previous trials failed (even when only E-mail was around; now we have the data silos by Twitter and Facebook that are additional obstacles).
Even after almost a decade, the well is still poisoned for everyone from Google's botched Wave rollout; if even Google (with the tremendous leverage of Gmail) was unable to pivot communication is this direction, who will?
https://en.wikipedia.org/wiki/XMPP#Connecting_to_other_proto...
In the same way, yes, you CAN link IRC to Slack and XMPP, but how many people have?
That doesn't mean that we should give up. One technology will finally get the right balance and take off, by learning from the mistakes of the precursors. Maybe that will be Matrix; maybe we will be another stepping stone in the journey - who knows.
One thing for sure is that big commercial for-profits like Google are absolutely not the best positioned to fix problems of cross-industry collaboration and defragmentation. Which is why Matrix attempts it as a neutral non-profit.