Shipping Rust code in Firefox
hacks.mozilla.org
hacks.mozilla.org
Overall the experience of using Rust for that project has been pretty great; the original requirements for a language were "able to produce static executables with no external runtime requirement, not prone to C-style memory safety issues, and accessible to a team with mostly Python backgrounds". That leaves quite a few options — Go for example — but being a Mozilla project taking the opportunity to use Rust seemed like it would align better with initiatives like those discussed in the article.
Apart from the general niceness of Rust-the-langauge, the experience of the Rust ecosystem has been really nice. Not only were there easy-to-install, well documented, packages to do much of the heavy lifting (http server, command line argument handling, etc.), it was also quite straightforward to set up travis builds using cargo to compile releases for linux64+musl osx and cross-compile releases for linux-arm7hf (added because someone asked about running on raspberry pi and it turned out to be trivial) and win64. The only wrinkle so far has been the difficulty of cross compiling to win32, meaning we might actually need to set up Appveyor or similar.
Obviously this is a much smaller project than the original post in terms of the number of users, and rather different as it is much higher level and could comfortably be written in Python or similar, except for the distribution requirements. However I think the use of Rust has been an unmitigated success, and I would be tempted to use it for a lot more projects that I would never consider writing in C/C++.
I wrote most of the initial implementation of geckodriver and mostly learnt Rust from reading the official book, writing some small patches for Servo, and working on this project (of course); pretty much the same advice you would get coming from any other language.
I just chatted with one of the other major contributors to the project who said that he was able to learn on the job by reading the existing code, looking up concepts in the book or via web searches, and asking questions when necessary; although some parts of Rust undoubtedly have a significant learning curve it is empirically possible to contribute to an existing project without a significant amount of upfront study. This arguably isn't too different from learning most languages, although you are going to need to read up on the rules around e.g. references and borrowing sooner than you would need to look something up if you took the same approach to writing your first Python.
The type system and borrow checker mean that the compiler will tell you if you're doing something wrong which is both a blessing and a curse; it can be dispiriting to get dozens of compile errors when you are getting started, but once you have satisfied the compiler it's possible to be more confident that your code won't break in ways that are relatively common in Python (e.g. missing or broken code for error handling).
So I'm not sure that I answered your question, but in practice it didn't seem to be a major problem. Of course in a different environment — one where people were less enthusiastic about learning a technology their colleagues were raving about, for example — you might have a different experience.
But at the end of the day the borrow checker is enforcing a relatively simple set of rules that you can learn and, with experience, intuit. So after a while the number of mistakes you make goes down, along with their severity. And there is payoff too in the ability to do things that would be impossible in Python and challenging in C e.g. write a copy-avoiding parser in performance critical code (not something too relevant to geckodriver, but useful for a so-far-prototype project to replace the log parsing on Mozilla's CI system — used to extract the reasons a job failed, and responsible for about 80% of the CPU time on that server — with a Rust alternative).
Edit: "Coming from" in the sense of bootstrapping great independent communities around these.
For example Gecko is gaining support for Servo's style engine so maybe CSS will be the first large component merged.
I wonder if Firefox will slowly become written in mostly Rust.
I doubt that will ever happen. Small parts of Firefox, yes, but the browser is enormous. I think I once read that even Servo, which is a showcase for Rust, has more C/C++ code in it than Rust code, largely because it uses Firefox's JS engine.
Not in the core project: https://github.com/servo/servo
(Note I have no doubt you are right. It would just be a useful metric.)
We've split the code up in crates so servo/servo is only a fraction of the story. Things in servo/servo are usually components which don't make much sense as something you'd independently use, and are very servo-specific and/or tightly coupled. But a lot of our Rust code is outside the tree, and any C++ modules we use are too.
Also, he vast majority of code in servo/servo is actually HTML/JS test code, vendored in tree from w3c/web-platform-tests.
Probably Github uses cloc. I downloaded the code (over 500MB, insane!) and cloc throws an error.
> and cloc throws an error
https://gist.github.com/steveklabnik/b4ede6f13c9d609edc61d74...
(using a gist since the output is huge, and see Manish's comment as well)
Though not all that rust code is written specifically for servo, and a lot of that C/++ code is winapi and skia. winapi is autogenerated, and skia isn't used by default anymore iirc.
I've never tried to find an optimizer bug in LLVM, but I have found more than one in V8, so I have some idea what I'm talking about.
[1] More specifically, this doesn't happen in situations where correctness of the generated code is relied on to provide safety guarantees. There are several websites that will compile and run Rust code for you, but none of them try to ban unsafe code, or filesystem/syscall access for that matter, at the language level; rather, their security model relies entirely on the OS sandbox the process runs in. Google's PNaCl uses (or used to use?) LLVM on untrusted code, but AFAIK the output of LLVM, the machine instructions, are still run through the NaCl validator, so getting LLVM to miscompile something wouldn't accomplish much. (NaCl also runs both LLVM itself and the untrusted code in an OS sandbox.)
But note that a lot of security problems are not in the jitcode but rather in C++ implementations of JS objects and in the compiler itself.
CakeML or Verisoft's C0 could be useful for assembly generation but not as sure there. Tough constraints in JIT. Edited to add Myreen's JIT that I just remembered.
http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=F58...
Now if only there was a way to isolate stuff at runtime (i.e. a sandbox) to better take advantage of operating system mitigations. I hope that with e10s now (almost) shipping we will start to see some progress towards this. IIRC Mozilla has said they would start to roll out security features (related to e10s) in stages. I'll believe it when I see it but it's nice to finally see this from Firefox since it's been far too long without a defense in depth approach to security from them.
Still wishes a safe parallel renderer to emerge so good luck to the team.
That someone is worried about these issues does not imply that they don't understand the value of telemetry data for developers. It's just a question of tradeoffs, and it's a decision the user should make.
I don't dislike error reporting in programs. I dislike it when programs call home without asking permission.
The restrictions on telemetry on release versions are pretty stringent. It's possible this was measured on release, I'm not aware of the details, but I suspect it wasn't.
(My understanding is based upon whenever I install Firefox it promts me to enable telemetry, which is disabled by default)
Firefox telemetry is opt-out on the Nightly, DevEdition, and Beta channels, but (mostly) opt-in on the Release channel. The opt-in telemetry from the Release channel has limited use because it is not representative of all users.
Here is a list of Firefox's telemetry measurements. Those that are opt-out on the Release channel are tagged "releaseChannelCollection": "opt-out". Adding a new opt-out measurement requires an additional privacy review.
https://hg.mozilla.org/mozilla-central/file/tip/toolkit/comp...
Again, ICBW here.
If it becomes something dependable, infrastructure-wise, like Java, this might mark the beginning of a more serious uptake.
Currently, I believe that this functionality is optional, so distros aren't required to actually ship it just yet.
For the produced binary packages (rpms, debs) it won't matter that much; although it will reduce build dependencies for source packages and give more consistent builds, even if your distribution comes with an _ancient_ version.
An analogue for rust would be something that detects the platform at runtime and then downloads a statically linked (?) binary that works on that platform. It can also exploit the ivy cache as the Scala compiler is just a library as far as SBT is concerned.
Isn't that a deal breaker? I would assume that downloading and checking in the binary for a whole compiler for 3+ platforms is untenable.
Also a few unofficial repos for rust/cargo on Copr https://copr.fedorainfracloud.org/coprs/fulltext/?fulltext=r...
Someone got a draft up last night [0], and discussion is happening on -devel. If things go well this may make it in F25.
Copr stands for "Cool Other Package Repositories" and is pronounced copper
In the past, we've explicitly added platform support for this reason, Windows XP comes to mind.
And until recently, as another example, rust didn't quite work yet on Solaris (amd64); thankfully, a community member stepped up and has been actively working on that port.
1) Making the "cargo vendor" story work better. rust-url has a bunch of dependencies, and you have to get them all in-tree.
2) More security review & planning. URL parsing is scary! And we'd want to ship & run it alongside the C++ one to check for places where rust-url is not fully web compatible, but there are major privacy issues in reporting back anything more than "1 failure," even for users who have explicitly opt'd in to reporting back data.
But the team is definitely working on both of these pieces and I'd hope to see it in the near future. No timeline / release number promises, though, right now :-)
I followed Rust at the beginning and I was pleased with the design goals of the language. However, I wasn't thrilled with the pre-1.0 documentation, and even after 1.0 there were breaking changes to the language.
Hopefully this is a sign of Rust's maturity. I'll have to look at it again:>)
EDIT: s/from/at/
So I haven't worried about anything per se, but breaking changes are mildly irritating to me. Without a doubt the utility value of some of these changes has been worth it (specifically RFC 1214, which was ironically one of the most major and definitely necessary). I'm not involved enough to be able to speak about other changes.
EDIT: I should also add that the software I work on professionally doesn't use any language that Rust competes with directly. Rust doesn't offer me professional utility so it's easy for me to complain about one thing and ignore all the benefits of Rust. If I were in a situation where I was contemplating using C++, Rust, or maybe Go, I think I would still choose Rust. It's far ahead of its competitors and gets a lot of things right.
> However, implementation of some Java SE 8 features required changes that could
> cause code that compiled with Java SE 7 to fail to compile with Java SE 8.
Especially in a statically-typed language, technically breaking changes are a fact of life. The key is to make them as minimal as possible, so that you don't struggle to update.
(Oh, and glad to hear you like the errors: we're actually working on making them even better. Elm is really leading the way here...)
The UX of compilers have been improving dramatically lately...
Though Steve may be talking about some newer work that I haven't heard of yet.
In general, Jonathan and Niko are working on it :)
I don't think it's fair to compare a language's breaking changes to a library's. I expect different levels of stability depending on the library, but with languages I expect the same level of stability. Rust __has__ been very good at dealing with breaking changes, yes. I'm stating that I don't like breaking changes in a language. Breaking changes in a 1.0 > language are something that should be exceedingly rare.
It might appear differently because we're very up front about any change that might possibly break any code, even theoretically. We don't make any changes that we think actually break code, except for blatant bug fixes.
Go has made changes post-1.0 that were more aggressive than anything Rust has done, such as changing the size of int.
>Go has made changes post-1.0 that were more aggressive
>than anything Rust has done, such as changing the size of int.
Even if this isn't true, Go's release note says:
>The language allows the implementation to choose whether
>the int type and uint types are 32 or 64 bits. [1]
All of the "breaking changes" in Rust have been of this form.
We're very up front about any possible breakage. When we talk about "breaking changes", we mean "a change that shouldn't be breaking, but could be breaking if code was relying on this bug/implementation-specific behavior". Those types of changes—changes that might cause breakage in practice but do not change the language definition—are often not considered "breaking changes" in Go. But in Rust we often call them "breaking changes", because we are concerned first and foremost about the practical considerations of changes we make, not just whether we are technically allowed to make them per the letter of the law.
Theoretically one should be using fixed-sized variables, but people may not realise they're relying on a particular behaviour. One of the whole points for using a language like Go over, say, C is that humans are imperfect and so having the computer assist is good, and this imperfection penetrates into all aspects of the software process. With a tasty name like "int" and being generally the default type, I'm sure a lot of code uses it when fixed sized types may be better.
> The only thing that changed was the implementation, not the language.
This is somewhat irrelevant: it still results in code not compiling or possibly changing behaviour, because people in fact have to use an implementation of Go, they don't write in an abstract idealised version of it. Rust's breaking changes are generally the same sort of thing (undocumented implementation details, changing the implementation to match the long-stated desired behaviour and bug fixes), but they still result in people's code not compiling or changing behaviour and so need to be handled as such.
Please don't include things like this in comments.
Changing the size of int. Changing methods to introspect the type of their arguments and do things differently. And so on.
> There were only 7 releases since go1 and I fail to find any breaking change in the language specification. On the other side each Rust release has a fat list with "BREAKING CHANGES".
Because we have a very specific definition of "breaking change" that is primarily concerned with the practical effect of changes we make. Go does not consider these changes "breaking". Using Go's definition (changes to the language definition), we have no "breaking changes".
If we were to change the size of int (something we will not do, by the way, due to the practical effects of making such a change), then we would list it under "breaking changes", even if we were technically allowed to do it. That's because we care about the practical effects of our changes, not just the letter of the language definition.
> Many Rust packages only work with specific Rust versions(i.e. nightly).
Because they are explicitly opting into unstable features that are carefully marked as such. We can't stop packages from doing that. Nor can any other language.
> The rust ecosystem(std lib, tools etc) is also way behind Go in terms of stability.
The parts of the Rust standard library that are marked stable have remained completely backwards compatible, in both interface and implementation.
> Why do you need rustup if Rust is so backward compatible?
Because it's nice to keep your compiler up to date and to target different platforms?
>> The parts of the Rust standard library that are marked stable have remained completely backwards compatible, in both interface and implementation.
This looks like a breaking change on a stable API. Am I wrong? https://github.com/rust-lang/rust/pull/28811
Rust doesn't have backwards incompatible language changes.
> This looks like a breaking change on a stable API.
No. It is a method addition. It is breaking only in the sense that code that didn't explicitly invoke the previous "as_ref" method might call this new method instead. It's the moral equivalent of:
type A struct { ... }
type B struct {
A
}
func (a ﹡A) Foo() {}
func (b ﹡B) Foo() {}
Compiler changes are the norm everywhere? I'm not sure what you're trying to say here.
> This looks like a breaking change on a stable API. Am I wrong?
Yes and no.
Rust's policy on breaking changes is that changes that can be fixed by properly qualifying an implicit path are not breaking. Otherwise, adding any method to anything would be a breaking change. In this case, you can use the UFCS syntax to disambiguate.
So it's an "allowed" kind of breakage because not allowing this means freezing the stdlib.
(Go doesn't have this issue due to lack of generics, overloading, and interface-based overloading. Edit: actually, go does too, due to inheritance, but that is easier to avoid and isolate. In rust you can always write client code that breaks if the stdlib adds a method, anywhere. This is true for most typed languages).
Anything that has the chance of practically breaking things is still run through crater (which tests impact on the ecosystem) and as you can see that PR had minimal impact.
Steve has pointed out that go did have this issue as it has a limited level of auto-deref, which it has changed in the past: implementations performed two levels of auto-deref when executing methods when the spec only requires one, the implementations were changed to only allow a single auto-deref: https://golang.org/doc/go1.4#methodonpointertopointer
Rust (and C++, because SFINAE, and many other languages), on the other hand, technically has a breaking change each time any method is added to any public type in the stdlib. It's always possible that the client lib was using a trait method of the same name, and now has to fully qualify the method call.
> Why is it not accurate?
I said it _was_ accurate. It's just not neccesarily complete. It's not complete because I'm only one human, and I have more important work to do.
That's not what I've seen from your comments. Instead I've seen some confused arguments about what "prose only" means (anyone in the PL field would consider both Rust and Go's documentation "prose"), combined with incorrect statements about both Rust and Go and a completely baseless assertion that Rust is "a language in flux".
The Rust maintainers would have considered this a breaking change. The Go maintainers did not. This isn't to say either side is right or wrong, just that they are measuring different things.
Additionally, the Rust maintainers have been exceedingly cautious whenever making these types of changes. They literally download, compile, and test all published crates to look for indications that such a change might actually break existing code. In the very few cases it has, they've worked with crate authors to incorporate fixes.
The very low bar Rust sets for determining what is a breaking change directly reflects the extreme regard they have for this issue.
To cross compile. To have quick toolchain updates. To get bleeding edge compiler improvements (e.g. speed) quickly. To test out new features. To help find bugs in the compiler.
One very common use case of rustup is to use clippy. Clippy is a developer tool which hooks directly into the compiler and uses all sorts of private APIs, an inherently unstable thing. It only works on nightly. Lots of people write their code to work on stable, but want to use this tool so they use rustup. Note that no language has a stable way of hooking into the compiler.
Very few rust packages only work with nightly. Care to provide some examples?
> Why do you need rustup if Rust is so backward compatible?
A staged release cadence with different levels of surety gives people the ability to play with features as they're developed to make sure those features solve the problems they're trying to solve (in the best way) by giving time for real-world experimentation and feedback. A feature can graduate from nightly-only to stable, and it then has a strong backwards compatibility requirement. The nightly experimentation period is valuable to get those features perfected before people can start relying on them more broadly.
(And dbaupp is correct that it's not always about testing; not all of the OS dev features are in stable yet, so nightly is the only option for that kind of project.)
>>What if I would like to guarantee this property for my own code?
I think that should be an exception if the backward compatibility is to be trusted.
Can you specify, in particular, what you think Rust is not doing that it should be doing?
Rust is doing exactly that.
> Although we expect that the vast majority of programs will
> maintain this compatibility over time, it is impossible to
> guarantee that no future change will break any program.
It's extremely similar to our attitude, and that of Java, etc:
> Of course, for all of these possibilities, should they arise,
> we would endeavor whenever feasible to update the specification,
> compilers, or libraries without affecting existing code.
This isn't a bad thing! My point is as I said below: every language includes some small breaking changes, even if the goal is to have very, very few of them.
It's not a breaking change according to Go's definition of breaking changes, by which Rust hasn't had any breaking change either.
> Rust doesn't even have a formal language specification.
Neither does Go, the spec is not a formal specification (and the first phrase of the intro states that it's a reference manual).
So does that mean despite a billion samples between those dates, all samples fell on just three dates?
* Firefox 45: https://telemetry.mozilla.org/new-pipeline/dist.html#!measur...
* Firefox 46: https://telemetry.mozilla.org/new-pipeline/dist.html#!measur...
* Firefox 47: https://telemetry.mozilla.org/new-pipeline/dist.html#!measur...
Also, 45 ESR doesn't include the Rust code on Windows (I heard that's landing in 48, if a Mozillian could please confirm that), so that's a large userbase to not have included in testing.
I think that would be an excellent project for someone, if you should both show all of Flash's capabilities implemented in Rust and that the result was safer it would be an excellent endorsement of Rust. It could also illuminate end user features which can never be made safe. Also good for the overall body of web knowledge.
Emphasis on "met". I've yet to come across a function that can be built in Flash, but not in HTML 5. In fact, I'm not using flashplayer at all anymore and I don't suffer. (There are a few video sites that are still Flash-only, but `mpv --ytdl` works around that very nicely.)
Come to think of it, this is actually not true. At work, I have to use Flash Player for exactly one thing: Adobe Connect. I wonder why they didn't move to Flash yet. ;)
This is great news for the language as it's going to help make sure a wider array of architectures are supported.
Awesome work!
As I read it, they are not shipping Rust but a component written in Rust.
> For this reason, Ralph Giles and Matthew Gregan built Mozilla’s first Rust
> media parser. And I’m happy to report that their code will be the first
> Rust component shipping in Firefox.It's just shocking to me this type of meta-discussion about the phrasing of an announcement headline is the bulk of top comments.
It took me, literally, years before I figured out 'check your brain at the door'.
"Shipping Rust code in Firefox" is what it's about. Though really "Shipping Rust in Firefox" means that too, it's just a bit ambiguous.
Assuming that they ship another language in addition to Javascript would not be completely out of the question. While it seems that with WebAssembly that is no longer necessary, who knows what Mozilla, in search of future funding, may come up with to open new markets. They also tried creating their own OS for mobile - shipping Rust to further its adoption for whatever to me at the moment inconceivable reason is no less improbable.
So yes, I wasn't sure what to expect from the headline - it said they are "shipping Rust" after all, so something crazy was certainly within realm of possibility.
I believe they have mentioned that a WebAssembly target for Rust compilation is something they are working towards, and IMO that would be the ideal way to deploy Rust for a website (if it's a compiled language, there's no need to deliver it uncompiled).
When WebAssembly is finished, stable, and widespread, I guarantee people will be using end-to-end Rust (among other languages).
Small nitpick, I know :)
We need that new moderator quickly! :)