What Rust Can Do That Other Languages Can't
robert.ocallahan.org
robert.ocallahan.org
For systems programmers, Rust looks a fantastic option. For line of business apps, it is inaccessible and the safety dial is turned up too far. But then I guess that is because Rust isn't intended to be a general purpose language for writing line of business apps?
I can see it being useful for writing platform neutral, gnarly code and offer easy hooks for popular, managed languages such as JavaSript, C#, Java et al to hook into.
Outside of systems programming, I wonder if Rust will be part of a silent revolution?
I frequently witness this rush to judge Rust as too complex to be used for business applications based on a few hours of hacking. For this to be a valid judgment, I would think one of two things would need to be true: a) Rust remains this difficult to use once you have experience with it, or b) most business applications are worked on for only a few hours.
In my experience, neither of those premises hold.
I believe that the rules of Rust not only give you exceptional performance without memory issues (as this article tries to demonstrate), but also that its rules engender a higher degree of correctness and 'well-factoredness' than most languages. I believe Rust is an excellent language for writing applications at every level of the stack when 'craftmanship' matters.
I believe part of the reason is that people have limited time to play with this stuff and so they want instant gratification.
Rust is not like that, but offers a much higher dose of delayed gratification instead. Once you familiarise yourself with the rules and get things working on the first or second compile, it means not only that you program appears to be working, but that it also passes a complex set of tests verified at compile time. When you become confident enough with Rust, you're basically as productive as with any other compiled language, with the added benefit that the code you wrote might actually be (re)usable afterwards.
With Rust, you put in some work up front and reap the benefits since then, with other languages you can feel 'productive' right away, but not be able to write production grade software for weeks/months to come.
Once you realise this, spending more than a few hours with Rust becomes an easier calculation.
My only gripe is that being a JS dev I'm spoiled with Node.js's gratuitous amount of community-written modules and finding out that a lot of the modules I picked for my projects in Rust are not actively maintained was a little bit disheartening.
There's an effort underway to 'bless' crates for common tasks, (ie tokio, futures for async, hyper for http etc.), so this will get better soon.
Honestly I'm more interested in stuff that involves some degree of heavy-lifting like real-time audio processing etc. for which JavaScript is going to be slow(no SIMD).
JavaScript on the other hand is full of quirks, but you get something to work quickly, even if it's buggy.
The huge advantage that Rust has over languages like JavaScript is that if it compiles, it's almost guaranteed to run (unless you have logic errors somewhere). I think that fact is greatly under-appreciated.
This. In my company, our Rust apps have a very small amount of tests, mostly integration tests written with Python. You need to test certain business logic, but mostly if it compiles, deploy all the things :)
You don't hear this often, but when you hear this, people a usually talking about either OCaml or Haskell. Nice to see Rust joining this club, too.
I often rephrase this as: "When switching from Python(/Ruby/PHP) to OCaml(/Haskell/Rust) you notice how many of your unit tests were really just type checks."
I'd say beginner tutorials should UNDERLINE that the use of unwrap() is just for the first prototype and you must refactor them out to proper error handling.
Or, to be more modern, there should be a linter rule for that.
I just wanted to say people are hating on the borrow checker, while they gladly invest time in learning quirks and I don't understand why :)
What will be interesting is if Rust proves to be, or can be made to be, a productive choice for programs where the runtime performance is not needed.
At the moment I feel like it adds so much verbosity compared to other ML-based languages that it just isn't. But I do love performance, even if I don't need it :)
In my experience JS is the worst at NOT working. Because when one part of the application fails, the entire PAGE stops working. Buttons don't work anymore, dropdowns don't work, nothing works.
And all of this is just one buggy line of code away.
To bring it back around, there are a lot of people who say things like "It's not as bad as it might seem at first" or "yes, at first you struggle, but then it becomes second nature, keep at it!" but I don't see anyone completely dismiss outright the difficulty up front. I've heard some people say that they thought it was going to be harder than it actually was, based on people trying to set accurate expectations.
Yes, sometimes I have the feeling people with a unnaturally low stress tolerance are the ones who comment the most :D
this is like a 10 second job in languages like C# or F#, but really weird in Rust.
And analogous things can come up with any graph that isn't directed.
enum BinaryTree {
Leaf(i32),
Node(BinaryTree, BinaryTree)
}
But perhaps there are some operations you are doing on the tree that are more difficult?Also, what kind of undirected graphs do you find yourself writing? I'm curious, because I rarely if ever find myself needing a graph.
recursive type `BinaryTree` has infinite size enum BinaryTree {
Leaf(i32),
Node(Box<BinaryTree>, Box<BinaryTree>)
} let mut basepath = PathBuf::from(path_arg);
for component in req.url.path() {
basepath.push(percent_decode(component.as_bytes()).decode_utf8_lossy().into_owned());
}
Granted -- each part of that makes a lot of sense, and I think it would be to Rust's detriment for things not to be that way. But when my focus has to be on rapid development rather than exacting precision, this sort of verbosity can get in the way.Personally, I have been using Rust anyway because I am trying to shift my development habits away from rapid development and towards exacting precision, and to this end Rust has been a godsend. But I can easily understand why this verbosity may push someone away. I do note, however, that verbosity is not the GP's complaint here.
Definitely the into_owned is unnecessary and could be replaced by an & before `percent_decode`. This ownership wrangling is something we'd like to address in the coming year.
Then I'm not sure about percent_decode. I think there could be a better convenience for that back and forth casting between utf8 and bytes that's going on.
The only real decision you're making here is decode_utf8_lossy vs decode_utf8. I'd like to think this could be encapsulated as two methods on the Url type.
The point I'm trying to make though, is that if the developer's focus is on rapid development (which I imagine most people in webdev are), then at the end of the day I just want to urldecode something, and not necessarily worry about the character encodings etc.
If you want to onboard people to your code base with no familiarity with the language or your system and have them be 'productive' (for some definition of productive) within the week, Rust is probably not the right choice for you. In my experience projects for which that is a high priority (and the languages which enable them) also deal with a great amount of technical debt.
But this is more or less why Rust isn't popular for LOB apps yet. If a business has a long-term, critical project, it needs to have relative certainty of its ability to hire programmers to work on that project, at least in case some of their current programmers leave the company, let alone wanting to hire more developers.
Right now there aren't many Rust developers, and training new Rust developers past the steep learning curve is hard. So employers, even if they recognize the value of Rust as a language for a potential project, have a hard time getting off the ground.
The good news is that the Rust 2017 Roadmap understands that this is one of the primary inhibitors of adoption and that the team is committed to making progress on this front. But it's why the language just isn't there yet.
The only gain you get going to Rust is really in the performance department. Rust is what you use if you want C-level performance combined with that safety, it might replace C and C++ as the OS, browser, high performance server and game development language.
For anything else I find it hard to suggest Rust as a serious option when there are many high level languages that seem like better answers.
I specifically asserted that Rust is capable of typing imperative code in a way that no other language is. ML-descended languages hide their imperative components behind Cells and Monads. Imperative languages have type systems which don't capture questions of ownership, aliasing, and shared mutability.
I found your comment to be not at all enlightening; all you've done is list other languages with type systems and then assert that I am wrong, suggesting you read my comment with a closed mind - in which case, why comment?
Maybe this is a misunderstanding on my part, but I found it of little concrete benefit for a lot of extra work. Perhaps my initial experiences with Rust have colored my thinking on it though.
Sorry to say this but a lot of unsafe code has been written in Java or C#, putting aside the NPEs and such. In Rust safety isn't only about eliminating SEGFAULTs and buffer overflows, but also preventing data races and most concurrency issues. It's common to add threading to cpu-bound code, without worrying about deadlocks or race conditions.
Rust also allows some very high-level code. Some of my favorite options out there:
- option (std): https://doc.rust-lang.org/std/option/index.html
- serde: https://github.com/serde-rs/serde
- docopt: https://github.com/docopt/docopt.rs
Some more features: generics (traits!), immutability by default, type inference, first-class functions, operator overloading, conversion between related types.
But again, it's not a generally good solution for CRUD-y business applications, webapps, mobile apps, etc where this sort of work is generally rare if it's present at all.
I definitely don't think it makes Rust seem like an appealing option. Between the limited community library selection, the verbose syntax and the borrow checker's extreme complexity to new users, I wouldn't suggest Rust to say, someone working on a web application currently because they will gain few if any advantages, but suffer many risks.
You're right that it isn't appealing to those who're looking for fast results and don't worry about maintaining a large codebase for years. For those like me, many of the features Rust provides solve issues we face on a day-to-day basis.
Yes the borrow checker is a pain for a while. Yes the libraries are still immature. Syntax is much more verbose than say Ruby or Python.
But IMO the benefits you reap from strong typing and borrow checking outweigh the steep learning curve. It's not just about memory management, it's also about controlling mutation (because mutation is a hard problem).
Besides, if I don't want automatic memory management, I could also use Ada or Freepascal, who are both reasonably safe, though not generally as safe as Rust, and are mature and have a well-established toolchain.
In Rust, on the other hand, once you gain some experience at least, you don't think about memory management that much, until the compiler tells you how your strategy is wrong, and then you adapt your strategy to cope with it. This is very different from C/C++, where typically your program just crash at random when you get it wrong, and then you have to figure out where or how it's wrong.
And yes, mature and well-established are nice features. On the other hand, I can tell you that modern tooling like cargo, even if not quite as mature and well-established, can also bring large productivity benefits with it. Dependency tracking and build systems are largely a solved problem in Rust.
I would argue that memory management in Rust is a much smaller burden then you might expect, bit it's still there.
Some responses:
- ownership / clone vs move = Valid concern in any imperative language.
- stack vs heap / when things are freed = Nope. Things are freed when you no longer use them. Are you upcasting to a parent "class"? Use `Box::new(value)`. Otherwise there's no cognitive overhead.
I come from C++ and I've written quite a lot of Python. Rust feels a lot more like a typed Python than C++, memory-wise.
C != C++, because in the latter you can use smart pointers - just like in Rust.
> Dependency tracking and build systems are largely a solved problem in Rust.
I wouldn't say so, especially since incremental compilation is still an experimental feature only available in nighty. And tracking objects dependencies across crates isn't even possible ATM.
I meant dependency tracking in terms of which libraries you need what versions of. Incremental compilation as a project in nightly is about improving compile times only, not about any functional change in build system/project management tooling. In this context, I'm not sure what you mean by tracking objects across dependencies.
Keeping compile times to a minimum is an essential task of a build system for me. That's why I wouldn't say that "build systems are a solved problem" in Rust.
> In this context, I'm not sure what you mean by tracking objects across dependencies.
When I make a change in one file in a crate, this triggers a complete rebuild of all the dependent crates.
That's the same thing as incremental compilation.
While that is what good C++ programmers do, you cannot (sadly) prevent others to program just like in C.
// We don't need unsafe here
or
print!("This is NOT unsafe");
So you'd want a static analyzer anyway.
With Rust, or any other systems safe language for that matter, I can be assured that any code that violates the type system is contained in such blocks.
In C or C++ written in C style, any string, vector, function call with out parameters, data structure manipulation can be hiding unsafe code.
It is quite different just having to review some well defined sections than the whole program.
However I do concede all of this fails flat when using third party binary libraries.
I do enjoy using C++ since C++ARM days, already back then many of the modern C++ practices were possible after templates got added into the language, yet enterprise "C compiled with C++ compiler" code is the norm in many companies.
Specially since many don't even bother to write type safe C++ wrappers and call C APIs directly.
Even though it's controversial, I think this is why it's a good thing Rust can't #include a C header directly (requiring bindgen to make a crate). The community has put work into making safe abstractions over the raw C.
grep /^unsafe(\s\{)|(\sfn)|(\strait)|(\simpl)/
Some variation of that, or four individual greps. I forgot how to use OR properly in Regex but each of those works individually. Tested on regex101 [0] against the following code: unsafe {
// Scary stuff...
}
unsafe trait Scary { }
unsafe impl Scary for i32 {}
unsafe fn danger_will_robinson() {
// Scary stuff...
}
That should catch all four instances that unsafe can be called in without any false positives. Since `fn`, `trait` and `impl` are keywords they won't change. Of course, any comments should be removed beforehand in case of a commented out unsafe block.I did this as a learning experience because I either regex101 doesn't properly implement /g or I don't understand capture groups. :D
That's an overstatement. You can use smart pointers in C++, but they're not at all equivalent to the usability of Rust's pointer abstractions.
For example, Rust is able to have the Rc/Arc split, allowing for increased performance in single-threaded code. You can't do that with shared_ptr, or at least, not in a way that's checked like in Rust.
1 - http://blog.skylight.io/rust-means-never-having-to-close-a-s...
In C++ the rules are enforced by optional tooling that most developers don't use.
At CppCon 2015, only 1% of the audience acknowledged using such tools at Herb's talk about the core guidelines.
The only difference is that you need to deal with whether or not things go on the heap, are refcounted, etc (but this decision is in the types, so you make it once and are done). Whether or not that's "manual" depends on your point of view. I consider it to be more along the lines of choosing guarantees.
(I know this isn't quite the same thing as liballoc on nightly)
The process is mostly automatic because you don't have to manually free the memory you use like you do in C. The difference is that the moment when the memory must be freed is determined at compile time by Rust, while a dynamic GC has to determine this at runtime, which uses more resources but also ends up being easier to use.
I've never worked with Rust, so I may be horribly wrong. Based on what I understood, it's a more than what you say: resources are freed at a time determined by the source code, no random involved, thus you get GC and RAII at once.
Not necessarily. D has a garbage collector and is used in games already such as Quantum Break. I do audio processing and had to limit GC usage in audio callbacks of course, but it was far from a showstopper.
Business applications do not care about safety.
> I believe that the rules of Rust not only give you exceptional performance without memory issues (as this article tries to demonstrate), but also that its rules engender a higher degree of correctness and 'well-factoredness' than most languages. I believe Rust is an excellent language for writing applications at every level of the stack when 'craftmanship' matters.
Performance is not one either.
And i am not saying it is good. I strongly support your case. But the thing is, these factors do not matters for most business out there.
ie. java is good enough.
My employer has a "line of business app" written in Rust. Here's what we're learned:
1. Rust does have a learning curve. It's easier to learn to use well than C++, in my opinion, but certainly harder than Ruby or Java. You really, really need to budget 2 weeks for turning an existing C++ developer into a productive and confident Rust developer. You can't take somebody who doesn't know the difference between "stack" and "heap", or who has never passed an anonymous function to "map", and expect them to bang out code rapidly in two days.
2. The library situation for business apps is just barely adequate. We need support for PostgreSQL, RabbitMQ, REST APIs, etc. In each case, high-quality libraries exist, but they're usually missing something. (For example: For PostgreSQL, we need to use both 'postgres' and 'diesel' to get the feature coverage we want. For RabbitMQ, we don't have heartbeat support. For REST APIs, we won't have good HTTP request mocking until the Hyper async stuff sorts itself out. And so on.)
3. The Rust code is just ridiculously fast. We're processing gigabytes of data, so we love this.
4. The Rust code is remarkably "trustworthy", and I'm not just talking about memory safety. If your program even compiles, it almost always works correctly (assuming you understood the spec). This has at least as much to do with Rust's 'enum' and 'match', which encourage people to enumerate and handle all cases, as it does with the borrow checker.
5. Rust code is very pleasant to maintain and refactor once written.
Overall, we like Rust, but we're mostly only using it where (a) we really want performance, or (b) we really want to make sure we handle lots of tricky corner cases correctly. Oh, or (c) for long-running jobs that take hours to run on real data, where compiler correctness-checks are a huge win. We'll probably gradually use Rust in more places as we get experience and the tools mature.
This is something I agree with strongly, and have had a hard time explaining to Java devs.
I'm looking forward to seeing how Rust (as well as its standard library) will be tweaked to become easier to learn (which is a goal for 2017). I don't think you'll be able to reduce these 2 weeks to 2 days this time next year, though.
You are one of the first people/companies using diesel in production, btw, which is really awesome! I hope we can help you get rid of your dependency on plain postgres sooner rather than later :)
If you're using Rust for business apps, you'll basically never touch 'unsafe'. For an extreme example from a weekend project, I wrote an MPEG 2 Program Stream / Packetized Elementary Stream / subtitle image decompressor, which obviously does a lot of bit-banging. (It's an ugly old format.) And I never touched 'unsafe' once. This is typical.
The only big exceptions are (1) working with existing C libraries, which isn't too hard once you get the hang of it, but which obviously requires 'unsafe', and (2) writing your own graph-like data structures from scratch, which you should avoid in favor of using an existing library like petgraph (https://docs.rs/petgraph/0.4.3/petgraph/). Basically, a doubly-linked list or cyclic graph is the worst possibly introduction to Rust for newbie Rust programmer. Start with something easier! I mean, unless you're the kind of person who tries to pole-vault learning curves, in which case, have fun.
Are you open sourcing this? I'd love to take a look
I mean, seriously, the book on unsafe Rust is called the Rustonomicon as if the memory model of the old default-unsafe-unless-you're-Dennis-Ritchie languages is some Lovecraftian horror brought on because of our thirst for the cursed FFI.
Meanwhile, Rust's damn near magic (at least for me, a non-functional systems mortal) affine type system, which can eliminate many logic bugs and massively increase code reuse, is all but forgotten. The type system is at the core of Rust's entire memory safety but you need to read dozens of scattered blog posts and projects, and the standard library docs, to start to get a feel for how to move from OOP to trait based composition and lifetimes. In aiming for simplicity, most docs fail to show in practical settings how you can use Rust's type system to redesign your architectures in natural ways that completely reframe the process of writing code, making it easier to port ways of thinking and love the borrow checker.
Sorry, that was a colorful rant. Teaching a new paradigm is tough and the Rust docs team have done a great job. As the language and community mature, better and better pedagogical practices will emerge and introductory material that approach the language from all angles will grow.
A good start could be a "Down the Rabbit Hole" red section sprinkled through out (like all those side boxes in textbooks and programming books) that emphasises that all of this safety can be negotiated with the compiler through the use of native (implemented in the standard library in Rust) types and links to the relevant chapters in a new window.
For example, when introducing ownership and borrowing rules there could be a quick blurb about RC, RefCell, and other types that let you pick and choose which guarantees you want the compiler to enforce and how "unsafe" is just a keyword to tell the borrow checker to buzz off for a second. Reemphasize it a couple of times in the same chapter and have a small chapter on how to debug corner cases. The point is to plant ideas in the reader, little factoids that may help things click or at least give them a starting point they may remember when they get frustrated by the compiler.
Besides, unwrap() is also terrible in this regard but it's the single most used feature in tutorial code. You need to teach someone Rust before they can learn idiomatic Rust.
The current book has a chapter with a title that's almost literally this: https://doc.rust-lang.org/stable/book/choosing-your-guarante...
I'm talking about scattering bits of that chapter all over the book, including the first concrete language chapter on Variable Bindings. There's over thirty chapters between "Ownership" and "Choosing your Guarantees" which I think creates massive friction for new developers who never even get to that chapter before getting frustrated and giving up.
The "Effective Rust" section of the book is in no particular order; each chapter is self-contained and can be read without the others.
That is one reason why the new book does not have this chapter, though. It's organized differently.
Is there any discussion on the genesis of the new book I can find? I'm curious to see the thought process behind the rewrite.
Because that's the wrong attitude, and would result in just recapitulating the problems of C. C programmers arrive at Rust wanting to write C, which is understandable, but if respond by telling them how to write C in Rust then they'll never see any of the advantages of Rust and frankly may as well not bother learning it in the first place.
> as if the memory model of the old default-unsafe-unless-you're-Dennis-Ritchie languages is some Lovecraftian horror brought on because of our thirst for the cursed FFI.
You're saying it isn't?
It's really at least half-true. As far as I understand the libraries of Rust itself can't have everything written in "safe" code.
The hard part is to win the potential programmers to use the safe code as much as they can but to understand where the limits are.
And that is the hardest thing to do as a lot of people prefer simplifications the kind of "everything must be done the X way" etc. I call that a "religious programming" approach and the programming world has it in immense amounts.
You can't express everything in safe code but wrapping an unsafe API into a safe one is relatively easy with Rust's type system. For example, typemap [1] allows you to have a unsafe hash map of types with any key type to any value through a safe wrapper that uses associated types to enforce the data structure invariants at compile time. Given a bunch of types T, for example, you can have a single safe map containing all of the mappings of Event<T> to Vec<T>, Handlers<T> to HashMap<T, FnMut(T)>, and T to &T, etc, even though transmuting random bits of memory is the definition of unsafe.
Extremely low-level code or code that uses FFI indeed uses unsafe, this in unavoidable, but most Rust code doesn't need unsafe at all.
Searching for "unsafe {" in the source files of Rust gives:
Matching lines: 2499 That's how many time some block is marked with "unsafe {"
Matching files: 663 That's how many files have it.
Total files searched: 8404 It's quite small source base.
But then I've removed the files in "test" folder. What I get is:
Matching lines: 1804, Matching files: 290, Total files searched: 1302
That means that almost every fourth rs file of Rust, when the test files are ignored, has "unsafe {" in it.
To quote you: "this in unavoidable." I agree, and nobody should pretend that it is avoidable. The reason people have to use C is exactly to do "hard stuff" on the "low" level you mention. Otherwise nobody would use anything but Python and equivalents.
What is called "low" level is actually the efficient algorithms and the efficient numerical calculations. And of course, once you implement that, you can keep the higher levels "safe." It takes a kind of honesty to be able to admit that much.
Disclaimer: I like Rust and I don't like extremists.
When deciding what should go in the standard library vs not, one qualification is "does this use a lot of unsafe?" If so, it should. This is so it gets more eyeballs on it.
The compiler and standard library will have much more unsafe code than most projects for largely this reason.
(And why is there a need to "defend" the idea that it isn't? I already pointed to my answer to that in my topmost post.)
Edit: to answer ""unavoidable" is a very strong statement" from the answer of this post: how could the very presence of "unsafe" parts exactly in the code written by the most informed programmers of that language be otherwise explained? I can't imagine anything more representative than the compiler and the libraries, except for the numerical algorithms, which are typically not included in the libraries of a lot of languages.
It's not possible to tell, this question is far too broad. And "unavoidable" is a very strong statement.
> And why is there a need to "defend" the idea that it isn't?
I'm not; I'm saying that the rust-lang/rust repo is going to have a much higher percentage of unsafe than a random Rust repo. Most applications have no unsafe, most libraries have little to no unsafe. Some libraries have a lot of unsafe.
That is, you chose the compiler as something intended to be representative, I don't think that's accurate.
Therefore I meant it as in; since a compiler is relatively low-level project, some unsafe code is (most likely) unavoidable, due to the task at hand.
The other poster seemed to take the unavoidable comment to mean something wider than what I was talking about.
A few things: you asked about it being unavoidable. Remember, the rust compiler and standard library is a near decade long project evolving extremely rapidly. This means that many things in the compiler aren't exactly done the best possible way as if you wrote things from scratch today. The standard library is shaping up. But a _lot_ of time and effort has been put in towards modernizing things. Years of your language changing as you implement your language is a recipe for lots of tech debt.
Furthermore, you asked about a lot of stuff that's _not_ in the standard library too, so it's not clear that pointing at the standard library is a good way to demonstrate how those things are developed as well. And while we have developed a decent set of stuff in the standard library for these cases, it's not like stuff just gets added anymore, either.
So, lots of reasons.
The interfaces matter a lot more than the implementations; we spent a lot of focus on the actual interfaces, stabilized them, and internal improvements are always landing.
And how many of those unsafe blocks could be removed and consolidated into libraries, if they are doing similar things?
The dial is certainly turned up further than most businesses are accustomed to. But if that business is doing something involving my credit card number, I think the threshold of "too far" ought to be somewhere beyond Rust.
I hope that Rust does get some substantial adoption for line of business apps, if only to give us some data on how much it really costs to deal with your buffer overflows in advance, instead of suffering the embarrassing outages and hacks that make the national news.
Most security flaws aren't things that can be directly checked by a compiler. There are some, certainly, and it's worth preventing those. But the vast majority is in application logic, which no compiler can protect. Competent programmers working in an expressive language is probably the best defense.
Basically the bugs normally claimed to be in application logic, that are mostly uncovered or extraneous corner cases are prime time for compiler verification.
I think there's much more to Rust than just a systems language. For anything that you need to have c binding for (say, a crypto library), it seems a very good choice.
I was also thinking of combining some high level, abstract language (Clojure/Haskell etc.) with perf-critical sections written in Rust.
For the same reason I also find it easier to dive into 3rd party Rust code and modify it. In general you don't have to worry about hidden side effects, global state, weird calling conventions and similar things with Rust.
Edit: With the caveat that the number of available libraries and SDKs is still low, but in the case of PHP that was the case until the last couple of years anyway after Composer and Packagist came along.
I came to Rust, from Java and I do remember Rust being hard, but not enough to quit. While I am no guru, the elision usually does job for me.
It wet my appetite enough to try it again in the future. I actually have an idea that needs a C++ wrapper. I'd like to try to use Rust instead of C++.
For most business apps GC is better option, than other memory-management options. I'm not seeing Rust in place of Python or Java, as well. But it could be used in some speed critical pieces, where C currently used, for example.
The same is true for Ada and Haskell. You point to a typical problem of people who are interested in Rust. They like the syntax and features but the compiler gets in their way to be productive in a short amount of time. That is frustrating.
The Rust community should not point so much emphasis on its beloved borrow checker like the Haskellers praise their monads. Both things are attractive only if their is real practical value in everyday coding. If devs need three to ten times more time than in C++ to develop something then they won't switch, even if C++ has no borrow checker. They could compensate this lack with verification tools, FramaC and the like.
I agree with Andreas Bergmeier's comment in the author's blog. Experienced developers may barely have problems with lifetime issues, in particular if they use a language with garbage collection which really makes sense for most business applications.
Rust would probably have much more convincing power if the community would go the way which Microsoft went to promote their C++ and C# compilers. They provided a very convenient Visual Studio IDE with excellent documentation.
People who need Rust know they need it.
They've been paying me to write docs for years at this point, and IDE integration is a major focus of this year.
struct X {
Y y;
ref Y getY() return { return y; }
}
This tells the compiler that getY() returns a referenced based on the implicit 'this' reference. It's available now (DIP25 was the proposal for it).We're now implementing DIP1000, which adds similar support for pointer values (DIP25 only dealt with references).
To see it in action:
alias Y = int;
struct X {
Y y;
ref Y getY() return { return y; }
}
ref Y foo()
{
X x;
return x.getY();
}
Compiling it yields: test.d(11): Error: escaping reference to local variable xWhile the post gives the example as a simple struct field thing, that example itself isn't where it actually becomes important. It becomes important when you have structs holding pointers to the interiors of other structs, and more complex reference-returning semantics. I illustrate this in a particularly complex situation in http://manishearth.github.io/blog/2015/05/03/where-rust-real...
But yeah, D is getting there :)
So would I. It isn't. But we're working on much more than this in order to provide guaranteed memory safety (although using a very different approach than Rust's).
It's just that Rust does not stand alone with this particular example.
It's not about returning a pointer to a struct. It's about the pointer being _safe_.
What if x was not allocated on the stack, but on the heap? y is part of x (not another independent object) so if x is freed/deleted then all references to y become invalid. Does D have a guarantee that this won't happen? (I think the answer is no).
This was noted in the post itself:
> Most other languages simply prevent you from giving away
> an interior reference, or require y to refer to a distinct
> heap object from the X.
struct X {
Y y;
scope ref Y getY() { return y; }
}
[0] https://github.com/dlang/DIPs/blob/master/DIPs/DIP1000.md class X {
public:
Y* getY(void) {
return &y;
}
private:
Y y;
};
}
`getY` returns a pointer to the struct field, this field is allocated as a part of the struct. There is no way (in C++) to verify that this pointer does not outlive the struct itself. In other words, your pointer may become stale, and the compiler has no way to check for it.Rust has a resource ownership model (borrow checker) that allows it to check at compile time that references do not outlive the data.
(1): See constructor number 7 here, which was designed for this purpose: http://en.cppreference.com/w/cpp/memory/shared_ptr/shared_pt...
Nevertheless, a better solution was pointed out in an answer on the blog and by humanrebar.
[1]: https://play.rust-lang.org/?gist=78e7af18c6fc7d37d9df96be63a...
[2]: https://play.rust-lang.org/?gist=2072a18b674685f20c9d1504a88...
Then the C++ translation above should probably be a const& in C++, which is acceptable. The returned reference could in that case be stored, but it's risky and a bad practice.
class Foo {
int bar;
int& getBar() { return bar; }
}
if you take ref of bar and Foo goes out of scope well fuck. It gets even worse, eg. : vector<int> foo;
int& bar = foo[i];
foo.push_back(1)
that last line can cause foo to reallocate the buffer and bar is then invalid, meanwhile that code will compile without warnings (well I'm guessing it will I haven't done C++ in a year :) )You can see the program crash
Here is a better example : http://rextester.com/MAJDQ8375
On VC++ it will return : Reference is 0 but value is 9
On clang it will return : Reference is 32579632 but value is 9
So let's say I wasn't mutating the 0 index, VC example would work fine at runtime and then clang would suddenly start giving garbage results. This has caused me week of debugging in a big code base.
Pun intended?
Are you are familiar with the substring problem in Java? Before 1.7.0_06, substrings where created using an offset into the buffer of the original string. Nice O(1) substring creation, the textbook example for benefits of immutability. Unfortunately, if you have long-lived, but short substrings taken out of short-lived, but long parent strings, the garbage collector won't find much to collect, since the substrings keep references to their parent buffers. This was no showstopper, but without immutability or in an environment with unmanaged memory it might very well be.
This Rust snippet is about similar cases of subcomponent sharing, just on a shorter timescale: the subcomponent is exposed in a way that allows the compiler to enforce that nobody keeps a reference for longer than it is guaranteed to be valid. You may look, you may even touch, but if you want to look again you have to ask again.
Other languages get by without this by garbage collection, by excessive precautionary copying or by simply hoping that the caller won't do anything stupid. Rust is unique in achieving the same runtime behavior as the latter but without the uncertainty. As a bonus, those un-keepable references cannot escape across concurrency borders so that in-place mutability becomes a lot more manageable.
As I understand it, if you have a structure that has a pointer to something (like a User struct pointing to an array of permissions or some such), and the memory pointed to will be freed when the structure is freed.
There is a case where code can keep a reference to this pointer around longer than the user struct it belongs to, so when code tries to use it it references an invalid pointer. This is bad. Rust takes care of this in some clever ways, so you cant have a pointer/reference that outlives it's parent.
Any garbage collector could (with some overhead) prevent invalid memory access when we're talking about an object with a pointer to another object. Rust is able to validate the correctness of pointers to offsets within the stack representation of an object.
(Rust also validates the correctness of references to heap allocated objects as well, of course!)
Go lets you return an interior reference. But Go uses a garbage collector instead of lifetime checks at compile time (which is an advantage or a drawback, depending on your requirements).
But the key point stands wrt Go; Go is among the languages that "can't do this." First, there is no way to get a reference to a field inside a struct in Go (the workaround is reflection which has runtime overhead, and therefore doesn't count) and if you could get such a reference the compiler would be forced to reference count the member via the GC to satisfy the Go memory model.
what?
https://play.golang.org/p/1OJ5U5ZwFJ
> if you could get such a reference the compiler would be forced to reference count the member via the GC to satisfy the Go memory model
no reference counting in go.
What aspect of the language in particular makes that impossible? Or is it?
However, this
- requires additional input from the programmer, such as lifetime parameters in Rust.
- C and C++ don't have ADTs which help working with lifetimes A LOT (e.g. having Option<Box<T>> as nullable pointer).
- will reject 99.9% of useful libraries and programs written in C.
- would be equivalent to using Rust.
// Rust
let v: Result<i32, i32> = Ok(1);
// std::variant (doesn't compile)
std::variant<i32, i32> v = 1;AFAICT the idiom for Result<i32, i32> is a pair of "numerical-result-if-success"/"errno-if-failure". Seems like a great idea, allows us to not overload a simple integer result and end up ignoring the failure path by accident. "Ok(n)" would get assigned to the "numerical-result-if-success" and "Err(m)" would get assigned to the "errno-if-failure". I think this means that "v.unwrap()" would panic if it had been assigned "Err(m)", which is what most people would want/expect.
AFAICT std::variant can't do overload resolution correctly when both of the union types are the same. I'll wager that if you use an aliased type it will still choke on this ambiguity. So if you wanted something similar in C++ maybe you'd have to wrap it so it's not POD?
99% of the C++ code out there wouldn't compile anymore and a lot of it would require significant re-architecturing in order to work with the new paradigm.
Or more likely the vast majority of the codebase would keep working in "legacy" mode which wouldn't play nice with the safe portions of the code (since you'd have to consider all calls to the rest of the code as unsafe). Same thing with any external library not updated to be "safe". It's like the python 2 -> 3 transition, except a lot worse because it's significantly more complicated and probably can't be automated easily.
The reason C++ was so successful is because of its backward compatibility (all the way back to C). I'm not sure it'd make a lot of sense to introduce such radical changes now. Especially given how incredibly huge the language is already.
Microsoft and Google are working on static analysis tooling for this, regarding C++, but the tools require you make use of the right types.
Also, like any tool not part of the language standard, its use is optional.
This is why writing data structures in Rust is considered an 'advanced' topic.
Under every language you use are some frightfully subtle data structures, and the easier they are to use (i.e., Python dicts), the more frightfully subtle they are under the hood.
(Since certain people have developed a knee twitch with the word "generic" lately, let me note that I mean generic in its most, ahem, generic sense, not specifically the programming language research term.)
Nope, not what i meant. Just a data structure suited to my very specific problem, optimized just for my particular case. That's usually much easier than solving the general problem in a generic manner. And that should be possible to do easily in a programming language.
I think the point is that it's harder to do this properly than it seems like it is, especially in a systems language. I've seen many "working" shared pointers in C++ that only worked if you used them in the way the original authors happened to use them. They wouldn't work if, say, you put a bunch of them in a std::vector.
So some of this is about what "works" means to different people.
Yes, you will on occasion need to implement your own datastructure to meet a particular need. If you do that you're going to have to sit down and think about it, work out the guarantees, ensure that it doesn't leak etc.. All the things you'd have to do in C, but you have to do it rarely.
There are some intermediate libraries (things like 'Cell') that allow for writing structures that wouldn't otherwise be possible.
I disagree. For everyone involved -- designing data structures should be a less frequent activity than designing flow control IMO.
While I would consider it part of a foundation of one's foray into computer science, it's not a common activity of most programmers. (Note the distinction between these educational/occupational terms.) Most people just don't do this often.
The fact that Rust enables very simple composition via crates means that we really can all use petgraph without much indigestion. The risk this enables is that we become dependent on critical packages like this. Hopefully we all got a glimpse of the left-pad awfulness and learned how to avoid the same problems for Rust.
Crates.io is never gonna have the left-pad issue, by design. npm also changed their policies after left-pad so it can't happen there again, too.
What is really needed is good documentation on this specific aspect.
If on the hand if the messaging is "if you find yourself reaching for unsafe, it is likely you need to rethink your code's design" and then also "common datastructures are hard to write without unsafe" there is some slight dissonance there (imho, ime).
I'm not looking for 'blessed libraries' of prebuilt datastructures. I want to code up datastructures (e.g quadtrees, graphs etc), and good guidance on when exactly unsafe is the only way to get this done.
I don't really want to pause in the middle of a project and have to spend serious amounts of time coding up a (custom/unusual) datastructure just because there is no way without using unsafe (etc).
Right now there doesn't seem to be good guidance on this aspect (writing datastructures in Rust) which is presented as an "advanced topic" and "the wrong way / the hard way" to learn Rust. (there is the linked list book, which serves as a starting point, and using Cell etc seems to work - I'm still struggling - which is fine, that is how people learn.)
PS: None of the above is really a criticism of Rust the language or the team. I think Rust is great, I'm just a bit frustrated with the 'build common datastructures being "advanced" ' aspect, but in the end it is probably just that I haven't been able to wrap my head around Rust yet
unsafe is highly discouraged, but designing the innards of abstractions like datastructures is one of the places where you basically need it.
Be aware that good generic datastructures are just as hard in C++. See https://news.ycombinator.com/item?id=13580418 . Naive datastructure implementations often have strict aliasing bugs or mess up on the destruction behavior. Using unsafe to write datastructures in Rust isn't particularly hard; it's just as tricky as it is in C++. Having to use unsafe is annoying, but it's a minor annoyance. The nomicon helps teach how you're supposed to use unsafe.
So far, it seems to me that a lot of 'textbook' data structures require unsafe, at least to translate in a mostly 1:1 manner. I think more novel patterns will emerge around non-1:1 translations in safe code, which may push at least some data structures out of the 'advanced Rust' territory.
I recently ran into this myself while implementing an order statistic red-black tree in (safe) Rust, which shouldn't be particularly complicated, but has been a good bit more verbose than C++ or Java would have been, primarily because I did a lot of manual bookkeeping to avoid unsafe. Still, it was enlightening and I very much enjoy using Rust.
[1] https://ricardomartins.cc/2016/06/08/interior-mutability
I agree, "people want to write code that runs and thats it" but there's a subset of folks who like to write their code in C/C++ and crank up the warnings, turn on ASan/UBSan, valgrind+electric fence, etc.
To be successful, Rust doesn't have to obsolete C and C++. It just has to attract a critical group of the next generation of systems programmers.
Rust has very little competition in this space (newly designed languages that can be used to write ISRs, bootloaders, OS kernels, etc). Many of the other popular languages strive to find some middle ground between C/C++ and Java/C#/Python.
At the end of the day, many of us are paid to ship features. For many industries, correctness and safety just can't compete with new features. It's the reason Lisp, Python, Ruby, and Go are so popular - you can quickly write programs which are both fast enough and memory safe.
The gaming industry, one of the bigger C++ consumers, will also probably not move to Rust for similar reasons: hitting a schedule is much more important than not crashing or not having memory leaks. You can always patch a game.
Not all errors, only a particular class of memory errors. Most of those memory errors are those that a GCed language doesn't have to worry about, and C/C++ have a bevy of tools to find those errors before they leave the developer's hands as well.
That said, Rust's borrow checker will also help protect against a class of shared memory mutation errors as well. Is the time cost of pleasing the borrow checker for every memory allocation worth this benefit? Probably something only individual developers can answer. My answer is, for now, no.
Isn't it a bit confusing to talk about pointer addition when the example struct is just
struct X {
y: Y
}
and thus the addresses &x and &y are (probably) the same. Maybe the author thinks that &y is obtained as &x+0, but.1. Allocate both X and Y in the same block of memory, maybe on the stack, perhaps in a heap. In inner loops, making calls to malloc is pure performance poison. Most common garbage collected languages are completely unable to do this, with the admirable exception of the C# and the other .NET languages (and possibly some others).
2. You want to access Y by reference, so that you can work with it without needing to make a copy. Again, this is very common in performance-critical code. You can do this in C, C++ and other existing "systems" languages.
3. You don't want to accidentally keep using Y once X (and hence the underlying storage for Y) is destroyed. This is a subtle and vile bug that can memory corruption, once-a-week crashes on production, and week-long debugging sessions, among other headaches. This is where C and C++ fail, and where Rust nails it.
Modern GCs use bump allocators and (ideally) combine allocations of objects and their subobjects where that is possible.
In really fast Rust code, my goal is usually to entirely eliminate all heap allocation in favor of zero-copy parsers. This can be tricky—especially for streaming I/O using buffers, because a single chunk of input might get split over two buffers—but Rust's borrow checker makes it possible to maintain references into other people's buffers without shooting yourself in the foot.
In the newer JITs that escaping allocation will be deferred until the last possible point in time.
This is actually a nice point about JVM languages, _new_ has defined behavior and can thus be elided if the effect is not observable. malloc (or new in C++, although I am not sure about the spec there) do not and eliding a call to new may break the language specification as function is not called that should have been.
I don't know what Rust does here, and what is allowed in regards to the new operator. i.e. must a local variable be on the stack or can it generate direct values in machine registers?
Local variables that are not explicitly heap-allocated (and don't have their address taken) can be on the stack or in registers, and this is true of both Rust and C/C++.
I find Dafny to be a more elegant and nice solution, since this language allows you to actually prove your code is valid when it's not obvious from the static analysis. But this might be because I'm somewhat more mathematically inclined (and i admit that Dafny is still by no means easy to use)
Ruby:
class X
attr_accessor :y
end
Crystal (full working example): class Y
end
class X
def initialize(y : Y)
@y = y
end
def y
@y
end
end
y = Y.new
x = X.new(y: y)
puts x.yBut other languages are Turing-complete too...
(Actually, upon further thought, it may be that rustc rejects all invalid Rust programs and some valid ones. gcc certainly accepts some invalid C programs.)
Other languages can return inner pointers; its not an interesting feature.
The only interesting thing is that it is both safe and has no gc... but really, this is just another example of safety in rust, via a relatively obscure code snippet.
If you want to pitch rusts safety features, we can do something a bit more interesting, surely?
You're absolutely right that there are much more interesting examples, e.g. https://t.co/HJV7G4n0PD. Doesn't fit in six lines though :-)