Rust at OneSignal
onesignal.com
onesignal.com
That diagram is the most interesting representation of a diagram I have ever seen. Do you have a tool to generate these for you, or was it hand crafted for this post? If it's a tool, is it available anywhere? If this is a built out tool, I would use this tool all the time.
On the article itself:
This is the most interesting type of posts about choosing Rust. Very balanced and aware that Rust gives a lot of upsides, but compared to other alternatives there are measurable downsides as well.
I think the biggest trade-off that they identified (and I've also gotten the impression from writing Rust on my own time, not for work) is the relative immaturity of the ecosystem, in as far as libraries available, and amount of developers supporting some of these libraries. What makes me hopeful, is that as a stable language (post 1.0) Rust is still relatively young. And you can definitely see things improving, with the community growing, more libraries propping up, and also gathering around specific libraries.
Thank you! We worked really hard on it.
> Do you have a tool to generate these for you, or was it hand crafted for this post? If it's a tool, is it available anywhere?
It was hand-made for this blog post. The diagram itself was drawn in a vector graphics editor and exported as SVG. Then, using web inspector tools to find which visual box corresponded to some line of svg markup, classes/ids were added to the items we cared about. Next came writing prose for each component. Finally, a bit of JavaScript and a CSS animation makes it come alive.
> If this is a built out tool, I would use this tool all the time.
Me too!
Which vector graphics editor did you use?
I seldom use SVG, because it is failed vector format. You still cannot guarantee an image will work properly across all browsers (desktop, mobile devices) or vector design tools.
PS or PDF have more chance of working across tools.
The best I've found is Mermaid [1], which has a Graphviz-like grammar. Unlike Graphviz, it can generate very clear, orthogonally laid-out diagrams with minimal effort, and it works better because it's designed for flowcharting (it supports classic flow diagrams, too). On the other hand, because it was designed for flowcharting, the automatic layout can generate less reasonably clutter when you have many entities with many connections. In particular, if you tried to generate the OP's diagram, I suspect it would accidentally lay out the boxes in a way that made some arrows cross — but I didn't actually try.
I've been considering writing a better tool for some time, but the layout is a hard theoretical nut to crack, and I'm not sure I would be up for it. The best idea I've had so far is to use genetic algorithms with some a fitness measurement that promotes cleanness (overlapping lines, density, horizontal/vertical extent and so on).
(not affiliated, it just does this very well)
Aside from efficiency (lots of diagrams for documentation or blogging), one benefit would be the ability to enforce a single theme (font, colours, etc.) across many diagrams, and push out new versions by tweaking the styles. Illustrator can't do that, and OmniGraffle is very weak at it.
I tried using Sketch, which has a support for reusable symbols, but it's pretty terrible at it.
I'm especially happy to see that they're using clippy. It's not a core piece of their infra like hyper, but it's still being used, which is nice.
How has it helped? Does it just keep the code clean or has it found mistakes and stuff like that too?
Loved the diagram, btw.
Clippy hasn't found any bugs in our code that I'm aware of, but it has been invaluable even so.
Clippy is really good at preventing bit-rot. When refactoring, it typically points out constructs that are no longer needed after other changes.
It's great for teaching how to write Rust effectively and succinctly. I've certainly learned a lot from it. One example lint in this area is using `for item in list` vs `for item in list.into_iter()`.
OnePush sets `#![cfg_attr(not(test), warn(result_unwrap_used))]` to make sure no `unwrap()`s are used in production code; all errors must be handled! `result_unwrap_used` has easily been the most valuable lint for us.
Yeah, clippy isn't solely for finding bugs -- a lot of the lints are about better style and cleaning up code, which often become useful after refactorings.
It would be cool if I could configure cargo to only use nightly Rust when invoking clippy, and use stable for all dev/test/prod builds. Do you happen to know if that's possible?
> if I could configure cargo to only use nightly Rust when invoking clippy,
This is
rustup run nightly cargo clippy
cargo +nightly clippy
Using clippy is not as straightforward as you put it because clippy and the latest nightly often are broken together. There would be much less hassle if it was a stable crate. Also no worries of backwards compatibility between stable/nightly.
- rg
The plan for "stable" clippy is to bundle it with releases, precompiled. It will still use the internal APIs, but from a user's perspective they just have to `rustup component add clippy` and it will magically work.
Whoa, that's the first I've heard about this. Very exciting plans!
We've got a good idea of how it will happen, but the infra stuff has to happen first -- nobody wants to complicate the existing infra if it's going to go away.
I don't have a clear timeline on that. A few months I guess.
On clippy's side we don't need many changes to make this work.
Given that clippy is a tool, not a dependency, folks aren't overly concerned about this -- it cannot infect dependency trees and only requires nightly for local development. Most people do `rustup run nightly cargo install clippy` followed by `cargo +nightly clippy` to run it locally without needing to change the default toolchain.
There are plans to distribute clippy as part of the rust "extended" distribution, so the stability issue will go away.
But that is clearly an awful idea. No one would like useful tools at the expense of "separation of concerns" and marginally increased compiler maintenance :P
You have the same issue with clang -- libclang is stable, but a lot of the plugins use the unstable clang plugin API.
This way the compiler has a marginal increment of maintenance effort but can remain flexible, change its internals at any point. It will also benefit users who for a multitude of reasons only want/need to use stable.
The 'belligerent refactoring' resonated with me - I have worked on untyped and statically typed codebases of varying sizes and when making huge changes there is no better tool for me personally than an expressive type system.
We're starting to use Rust for static binary quasi-safety-critical applications, but communication/control-plane stuff like what's described at OneSignal we've found is much easier w/ Erlang.
Where'd you find them? I'd _love_ to find a vein of them to mine over time. :-)
- Rust is still a really young language and most of us just use it as a hobby and don't write Rust for a living, but many would love to. And for some of these people (I'm one of them), been able to use Rust at work can be a good reason to join you. - IMHO Rust is a good language for recruitment because - it has a high barrier to entry - you can't just fake knowing Rust: if a candidate doesn't understand the ownership system really well, he won't even be able to write a 100 lines code sample.
To get in touch with potential candidates you could post on /r/rust, and I'm pretty sure it will be well received since many people are eager for Rust job opportunities.
Of course if you need someone with 3 year+ professional experience with Rust, your choice will be a bit narrow ;)
It's free; how does OneSignal make money?
We make money by using the data we aggregate to improve web and mobile experiences. We also offer custom solutions to enterprise clients.
Right now my biggest pain, are the compilation times, specially seeing cargo compiling multiple times the same crates instead of reusing the already binary compiled libraries.
Unless there is some magic cargo incantation I am missing.
"a bit of catch up" is one thing, not mentioning most powerful IDE plugin is another.
1 - Get Project A
2 - Crate X gets compiled as one of the dependencies
3 - Get Project B
4 - Crate X gets compiled again as one of the dependencies
I usually see this when compiling all the VSCode related projects during new Rust releases.
Could the reason be that Crate X on Project A and Project B isn't the same version?
Where are the binary rlib visible to all cargo projects stored? I could only find text versions zipped together.
I haven't checked this deeply yet, just seeing "Compiling crate X" multiple times isn't fun.
On other AOT compiled languages that I use at work, I can reuse binaries across projects, including the languages Rust intends to replace.
It is a hard sell, if using Rust means having to spend more time compiling than even with C++ (specially now that modules are finally coming, at least for us with VC++).
To me it seems FOSS undervalues the value of binary libraries in the enterprise, or how they get used in many corporations or companies selling components.
Rust improved safety is a good thing™, but not at the expense of continuously seeing the same libraries being compiled over and over, vs reusing binary artifacts across projects.
Even with MIR, Rust will never compile faster than C or even C++ if dependencies on binary libraries aren't supported.
Right now I can compile my C++ projects on Windows, and .NET Native ones, faster than when updating Rustfmt, racer and Rustsym, after each new Rust release.
VS2017 will bring the new linker, and the ongoing MS work for C++ modules (now a TS).
Rust won't win the hearts of C++ developers if it doesn't allow for binary dependencies, and doesn't compile faster.
What? I've almost exclusively heard the opposite: Go makes error handling too in-your-face. People don't like writing "if err != nil" constantly.
>Finally, the developer leading the effort had experience and a strong preference for Rust. There are plenty of technologies that would have met our requirements, but deferring to a personal preference made a lot of sense. Having engineers excited about what they are working on is incredibly valuable.
I wish it were more acceptable to admit this instead of having to half-assedly argue that your favorite language just so happens to be a perfect fit for the problem. There's no shame in simply using the language you're most proficient in. Practically speaking, language choice is only a problem for large teams or for projects with language-specific requirements.
Do you actually have to look at the value you err, or can you just pretend like it's not there? (genuinely asking, I thought it was the latter).
> I wish it were more acceptable to admit this instead of having to half-assedly argue that your favorite language just so happens to be a perfect fit for the problem.
This is one of the reasons we felt it important to mention. Rather than demand purely technical motivations, we choose to acknowledge the human aspects of software engineering as well.
> > What? I've almost exclusively heard the opposite: Go makes error handling too in-your-face. People don't like writing "if err != nil" constantly.
> Do you actually have to look at the value you err, or can you just pretend like it's not there? (genuinely asking, I thought it was the latter).
While Go may not have all the capabilities Rust has at forcing error inspection, it really isn't in the class of languages that "make it too easy to ignore errors". I agree that a lot of it is convention, but multi-value returns, usage of named variables and good practices in the base libs (flowing upwards) can't place it in the realm of easily ignoring errors.
result, _ := someFunc()
_ = []interface{}{ err1, err2, var1, etc }
Nice writeup btw. Did generics/GC play any part in the decision making process?
Practically speaking, Go makes it hard to accidentally ignore errors if the function you're calling normally returns a value and easy to accidentally ignore them if it doesn't. For example, Go will force you to handle the error when unpacking the result of os.Open(), but not the result of os.Chmod(). There is a popular errcheck tool that does simple static analysis to catch this.
By contrast, rustc warns if you drop any error anywhere, and this warning ships on by default with the compiler.
The conversation was specifically about ignoring errors. In Rust, errors are conventionally reported using Result hence it warning on unuse — AFAIK the only type opted into that by default, its variants being specifically called Ok and Err, and its currently exclusive support in `try!` and `?`
> And even in this case you can just add "let _ =" before call to suppress warning.
You can even disable the warning entirely, can you imagine that?
Incidentally, you can use that exact same feature to ignore errors from value-returning functions (where you need the actual result) in Go as well, so that's not exactly a slam dunk; and it is an explicit ignoring of the error rather than an implicit one which can (and often is) a mistake rather than a conscious decision by the developer. That is, there is a large difference between
_ := os.Chmod(…)
os.Chmod(…)
In Rust, the latter is will warn by default because fs::set_permissions returns an io::Result<()>.
I like Rust's way to handle errors with Result, but I hate your tone, so discussion is over.
the signature ;"fn risky_shtuff() -> Result<Response , Err)"
Rust guarantees to you as the author who ever uses this function HAS to handle both cases (even if it's an unwrap).
However Rust doesn't stop you form making functions that instead return Option ,bools or even strings ,but you'd be actively working against very strong convention.
And for me it was one of weak points of this article. "We use Rust because our engineers want to play with a new toy" is a weak argument and soon will be used by opponents of Rust as their proof.
How is this not what serde does internally? I could certainly imagine a nice library for a dynamic language that has a roughly equivalent API.
BTW - I have all the motivation now to plunge into the rust-world
Rust should really address the top problem with C++ which is slow build times.
My VC++ builds still compile faster and that is just using the latest VC++ 2015 improvements without the modules work being done by MS.
Incidentally, a module system will help with build times, but that's only one of the benefits.
I'm curios: this looks like a perfect use case for Apache Kafka.
Did you evaluate it?
why not something like pgbouncer?
this is similar to the java approach of building connection pooling into the application itself using jdbc.
I'm actually very intrigued on if you are seeing actual benefits using r2d2 ... or would using only pgbouncer make it just as efficient.
There is huge advantage, not mentioned in article: when code has a lot of work with multithreaded tasks, race-conditions bugs will bite you sooner or later, and they will bite very very painful. They are hard to detect, very hard to reproduce and it takes long time to fix them. Rust is immune to race-conditions, and it's really awesome and enough reason to use it in apps like this.
Safe Rust guarantees no _data races_, but not race conditions in general.[1]