A Guide to Porting C/C++ to Rust
locka99.gitbooks.io
locka99.gitbooks.io
You don't want to convert entire applications at once, you want to do it a source file or module at a time. This means incorporating the Rust toolchain into the build of the C/C++ project Makefile, or etc., system.
It is not difficult, and will increase confidence in the final product. This can even be used to test incremental conversions in production, rather than having any doubt about the entire conversion.
This is an awesome start for a great manual on the area. I'm very inspired by the experience I had in doing this, and hope that we do more.
Can you tell us a bit about your experience with Corrode? The Readme itself states that it's not expected to work on "real" code, and there are open issues like this: https://github.com/jameysharp/corrode/issues/52 titled "Arrays are broken!" in which Corrode's developer wrote half a year ago: "The current hack gets arrays translated correctly in function arguments, but gets them wrong everywhere else."
In your experience, does it work better than this makes it seem?
(To be somewhat fair, the issue does have detailed discussion, and partial fixes seem to have been merged, but it's still open.)
https://github.com/jameysharp/corrode/issues/109
Most of the issues I ran into were not exactly with the corrode tool, but with compiler options, i.e. passing the DARWIN_C_LEVEL flag for macOS (my main dev machine). Actually, the library I hacked on was only built for Linux, I spent a lot of my time converting the C to multi-platform, not because it was necessary, but because I didn't feel like running a VM.
What I found is that corrode produces mostly compilable code. I had played around a bit with FFI from C to Rust before, so the task of integrating a Rust library into a C codebase was already familiar to me. I would love to see a guide for doing that in corrode though, like an example project, for others. If I have time I might do that. One really cool thing, is that the Rust compiler detected some undefined behavior in our C code, which I filed a bug for to our team (it was an uninitialized stack variable, passed as a pointer to a function; in practice it would never actually rely on the undefined behavior since all the fields of the struct were set in that function, but it's still poor form).
There are some things which you will need to massage, but I found in general that it wasn't much. I had a problem with null function pointer creation that I need to go back and look at.
I think my biggest issue, is that every conversion produces unsafe code in Rust. It's left up to the developer to figure out how to convert it to safe code. I think this is ok, but unsafe is still dark arts for me in Rust (meaning that I don't completely grok the best practices right now). I need to go back and reread the Rustonomicon: https://doc.rust-lang.org/nightly/nomicon/ , as there may be more details there than the last time I checked it out. I would love it if corrode could try and produce more safe blocks of code, but I also understand why it's this way.
Anyway, I think it's an awesome tool, works 95% of the time, and you can probably easily come up with some standard techniques for going back to fix up the commonly incorrect cases.
I'll reiterate this: integrate the Rust code into your C build right at the beginning (I used Cargo + staticlib), I even made this optional in my setup, so that you can flip between the two implementations easily for testing. I set up a make recipe which converts the C src to Rust with corrode (to capture all the CFLAGS needed), and that will only run as long as the .rs file doesn't exist. I found this to be very powerful for verification. I'd highly recommend corrode at this point, but don't be misled to believe that you won't need to make changes to the produced code.
Edit: I should mention that to get all this setup, with all the C library fixes in macOS included and the Makefile integration, it took me 6 hours to get setup, with about 10-15 minutes spent actually making the produced .rs file compile.
One warning I can give you based on recent HN posts:
Embrace Rust fully. Many expect Rust to be a "safe C++" or something of the sort, and try to port not only the code but their way of thinking into rust... which leads to pain and frustration.
You seem to have lots of experience so maybe you already know that, but I'm pointing it out just in case ;)
I'd say C best practices probably translate better to Rust than C++, but with solid systems knowledge, as long as you don't fight the language, and instead try and embrace it, I think it's fine.
I think a lot of people want to bring old patterns to Rust, some translate, some don't, but fundamentally the memory system is basically the same.
The "corrode" translator apparently just compiles C to unmaintainable, unsafe Rust. Raw pointers in C become unsafe raw pointers in Rust. That's probably not too useful. See [2]. Now you have heavy technical debt. Also, losing all the comments during translation is cruel to later maintenance programmers.
Good inter-language translation for programming languages is very hard. Usually, you lose the idioms of both the source language and the target language. There are few, if any, good systems for this.
To do this well, you need verification-like analysis of the source program. You need to know if a pointer can ever be null. You need to know how big a passed array is. You need to know if something has single ownership. If an analyzer could look at a C program and answer "yes", "no", or "don't know" to those questions, with a reasonably low percentage of "don't knows", then you could do a translation to proper Rust.
You could brute-force some of this by running the Rust compiler from the translator. Start out with everything being a single-owner reference, and compile. Anything that generates a borrow checker error gets turned into a reference counted item on the next try.
[1] https://locka99.gitbooks.io/a-guide-to-porting-c-to-rust/con...
The point is for it to be a first step in conversion, with large swathes of code being easy to make safe.
The issue with idioms is a real one.
But still, it gives you a good path for incremental conversion which IMO is pretty valuable.
This isn't a syntax problem. It's a semantic problem.
The rust code though, can be converted to pure Rust. A strategy I use and most other FFI code I've seen, is to have a shim layer, which is unsafe, that calls through to safe code. And vise versa for calling out to C. The function definitions don't need to be unsafe, just the direct calls to C and conversions to C types.
I never said it was.
You can still make trivial semantic transformations on code by hand or by using a tool (corrode could do it itself, too) and make it safe. As bluejekyll said it's a bit tricky since you can't guarantee anything about the calls until you port those too.
Rust is much more robust about catching those cases and verifying that you don't have objects outliving their references.
That's something that the programmer needs to specifically ask for, and I'm not aware of any C++ compiler which doesn't throw a bunch of warnings regarding an object's scope and lifetime.
That means your point boils down to arguing that if a programmer really wants to, he can still shoot himself in the proverbial foot by writing C++ code, but in order for that to happen he needs to have a death wish and intentionally avoid all warnings and safe practices to begin with.
#include <vector>
int main()
{
struct Foo {
int bar;
};
Foo* bar;
{
std::vector<Foo> foos;
foos.emplace_back(Foo{1});
foos.emplace_back(Foo{2});
bar = &foos[1];
}
bar->bar = 2;
}However, I never spent more than 10 minutes fixing that.
Debug version of MS C runtime fills freed heap memory with a magic number 0xFEEEFEEE. You’ll get exception in runtime, and you’ll immediately have the idea what’s wrong with the code.
I've seen 5 minute fixes and things that took weeks to track down(yay for non deterministic repros).
Technical support guys were not that happy with the customer regularly calling them to know when the fix would be delivered.
Have other war stories about tracking down leaks as well.
Most commercial libraries are written in C or C++. The reasons include performance, smaller runtime, and ease of use (all OS kernels are written in C, so the majority of languages already feature decent C interop). Even if you code Rust, you won’t have that compile-time checks inside proprietary third-party libraries.
FWIW it's getting pretty common in the Rust ecosystem to wrap 3rd party libs with proper lifetime annotations with opaque types when possible. Things like OpenGL, JNI bindings and whatnot can still benefit from lifetime annotations the compiler provides.
Rust proponents often overestimate practical impart of those C++ safely problems fixed by Rust.
At the same time, they often underestimate or ignore Rust’s problems caused by the same safety thing. The main of which is, it’s hard to implement some widely used data structures in Rust. Here’s my old thread where I discuss the topic in details:
Are there categories of UAF that would not be caught with valgrind / asan? Or is it the case that the test suites do not have sufficient coverage to exhibit the issues?
Requires manual annotation and isn't nearly as systematic and comprehensive as the Rust compiler, but I've found threading bugs with this in existing code that I was about to modify.
No, it's not.
Use-after-free is very easy. Just push onto a vector while you're iterating over it, to name one obvious example that comes up a lot.
We were testing our application the night before a big demo for our client. Ours was a C++ OpenGL application with several "phases" the user could enter from a "main menu" type screen. Our demo explored these phases in a pre-planned order. All went well, until somebody decided to try doing the demo in a slightly different order. Once the tester visited a certain phase of the application, the thing crashed and burned. The stack trace was littered with some incomprehensible container-implementation-detail junk. Debuggers weren't pointing at an obvious sign, other than that something was dereferencing a garbage pointer.
We noticed that if we went back to the original canned order of operations (this was a demo, after all), the application didn't crash. But this still made us nervous. We found that we could sometimes swap a few phases early in the application's run with no ill effect.
Here's part of the class responsible for some of the higher layers of the stack trace, from one of the lower level libraries we build all of our applications on (this is important because, for hysterical raisins, we can't source-step debug those libraries in production). Can you spot the bug?
#include <map>
#include <string>
#include <vector>
struct Message;
class MessageCentral {
public:
void HandleMessages() {
// messages were collected somewhere and organized by their queue.
const std::vector<std::string> queues_with_new_messages;
for (const std::string &queue : queues_with_new_messages) {
const std::vector<Message> msgs; // = messages_for_queue(queue);
const auto &handlers = handlers_[queue];
for (auto &h : handlers) {
for (const auto &msg : msgs) {
h(msg);
}
}
}
}
void RegisterMessageHandler(const std::string &queue_name, std::function<void(Message)> handler) {
handlers_[queue_name].push_back(handler);
}
private:
std::map<std::string, std::vector<std::function<void(Message)>>> handlers_;
};
(oh, by the way, we couldn't use runtime memory sanitizer tools like ASan or valgrind because the application linked to CEF. In retrospect, we could have dropped in a fake CEF that didn't interfere with ASan or valgrind, but that would have required build resources we didn't have available in the frenzied night before a big demo.)
I assume one of your handlers invokes RegisterMessageHandler invalidates the iterators to handlers_[some_queue_name]. As such, the following line exhibits UB:
for (auto &h : handlers) {
EDIT: I'll also note that using a debug build of the SC++L with iterator debugging enabled should give you a useful diagnostic here - given a modern SC++L implementation. Whether or not your application runs well enough, to actually get to the point of being able to reproduce the bug with iterator debugging enabled, is another matter entirely of course.
EDIT x2: I'll further note I'm cheating here by having encountered and fixed this same bug on a different codebase.
What I have learned since learning it in 1993, using it including teaching the language to first year students, is that any safety feature that isn't imposed by the language instead outsourced to best practices or external tools tends to be ignored by the majority.
Also, this is just me, given experience writing code in C++ and Rust, I find Rust to be a more ergonomic language (YMMV).
more info: https://blogs.msdn.microsoft.com/vcblog/2016/10/12/cppcorech...
This is what I mean in another thread about outsourcing safety.
Then you cannot use Rust and must settle for lack of safety. (A profoundly silly question -- if modern C++ is not an option for whatever reason, then Rust is doubly so.)
Just because the only existing Rust compiler uses LLVM, it doesn't mean another implementations will not surface.
So hypothetical the OS not targeted by clang, can still have a Rust compiler on the OS SDK, offering all safety guarantees from Rust.
For example, there are OSes and hardware architectures not supported by clang that have Ada and even Java AOT compilers.
If you want an option that works today without nonsense as you put it, then I would pick Ada over C++ for secure critical projects in such scenarios.
Companies like LDRA do exist, because just using clang isn't enough.
http://www.ldra.com/en/software-quality-test-tools/group/by-...
Also a reason why HIC exists http://www.codingstandard.com/
As a long-time user of C++, I find Rust to be more appealing in many ways. I don't think "C++ is dangerous" is a good approach to advertising Rust to the C++ community.
A good way to see what C++ community cares about, in order to make Rust appealing to them, is to see the work from SG14 (games and HPC) on ANSI C++.
That is the type of crowd that Rust should target, and they care about performance without belts above all.
Working with teams on typical enterprise environments is another total different story.
Just last week I had to do an acceptance review of a C# application for a customer, I surely wouldn't want those devs touching C++ as well.
If you're writing Firefox or SSL, a memory mistake may be a vulnerability and a serious issue. In games, hpc, hft, etc, a memory mistake is almost always just another bug.
Would be great instead to see more examples of how e.g. destructive move in That allows generating better assembly.
In HPC a memory mistake is yet another way of producing corrupt results for writing that paper, that after all is wrong.
In HFT a memory mistake is yet another way of producing corrupt values, leading to the automated trading bot selling items way lower than they should have been, loosing millions in the process.
I'm not a game developer so I can't judge your estimation of bug severity there but it seems highly speculative to say the least.
On the other hand your comments vis-a-vis HPC and HFT show that you have very little domain knowledge. Memory corruption is far more likely to crash a program, or to cause results that are wildly wrong. Wildly wrong is not so bad; any scientist worth his salt will question the results and see there are issues.
Your comments on HFT are even worse. Exchanges will not accept "selling items way lower", placing orders too high or too low produces rejects. And even if the order is accepted, it simply crosses the book at the best prices on the book. If you want to write about how you really lose millions, read about Knight Capital. Nothing to do with memory.
In both HPC and HFT, subtly wrong is a much bigger problem than wildly wrong. And memory is much less likely to produce subtly wrong unless someone is deliberately exploiting it. Again, it's not a matter of this never happening. It's just that it's rare in comparison to all of the other ways in which you can make mistakes, and particularly subtle mistakes.
This really seems to be one of the problems with Rust. Most people involved seem to be very firmly in the classical SV-esque tech world, working on browsers or core infrastructural components for the internet, or web servers, or what not. Chrome bugs and heartbleed are the examples du jour.
The problem is that this is not the main area of use for C++. The dominant industries for C++ are finance, video games, and HPC, along with some more demanding programs on mobile, or (rarely) on desktop. People like you who lack domain knowledge about the main areas in which C++ are used, stomping their foot and insisting that memory problems are central and that Rust is valuable, does not really achieve anything.
One example isn't about the destructive aspect: a move in Rust is always a memcpy. Since there's no move constructors, there's no way to run arbitrary code, so it's always a memcpy. The optimizer can also then elide this copy if it isn't actually needed.
Specifically, I think this work should be regarded as a "work in progress." :-)
I think you're point is completely correct though. You can write very safe code today using modern C++ features.
---
Reply to moomin:
The type system's intrinsic expressiveness limits the kinds of verification tricks you can “teach” it.
That's the primary reason Java took off. If you know C, you can write working C++ and Java programs straight away. For rust, we have to stop and relearn how to apply the logic (thus slow/no adoption).
And, I'm really sick of the constant HN sales pitches for rust. Make the transition easy and rust will stand on its own/sell itself (or not).
Criticizing it because C/C++ programmers can't just drop into it and start productively writing production-quality code is very short-sighted, IMO, and it's amazing how much it happens around here.
So now that even them have got the AOT memo, and Go and Swift also exist, I think the 2017's focus on productivity is very important.
The work done in the borrow check is incredibly and has already influenced decisions on the D, Swift and C++ communities.
Yet they all agree that they don't want to make it as hard as Rust.
Chris Lattner mentioned the feature in Swift should be exposed to expert programmers doing kernel and driver related programming.
I imagine Rust's image should be better than a language from experts for experts.
If the yardstick of “productivity” is how fast you can write incorrect programs, I'm not terribly interested in “productivity”. I'm sick of being a defensive user, having to refrain from probing how systems work in undocumented and likely unforeseen scenarios, lest I completely break them in unrecoverable ways.
I'd rather see an axiomatic semantics for unsafe Rust, so that I can formally prove that my unsafe Rust code doesn't break safe Rust's integrity guarantees.
> So now that even them have got the AOT memo, and Go and Swift also exist
Support for running in an unmanaged environment is a red herring. If you want to manage resources correctly, you have to care about ownership and lifetimes, regardless of whether you are doing systems programming (user programs manage resources too), need AOT compilation (JIT-compiled programs manage resources too) or need a GC-less environment (GC doesn't work for resources that need to be reclaimed eager and deterministically).
> I imagine Rust's image should be better than a language from experts for experts.
I have no idea. I'm not a member of any community (not a “Rustacean”, not a “Haskeller”, not anything), and I have no personal interest (financial or emotional) in any specific technology becoming popular. I only care about technical facts. Are facts for experts only?
Which is why usually there are other language or runtime features for deterministicaly managing resources besides just the GC.
The question is how to keep the way of Rust memory management, while e.g. making it easier to write cyclic graphs, or reducing even further the cases where one has to explicity type lifetimes.
Other than what Rust (and its research ancestors) uses, I haven't seen any alternative that isn't fundamentally broken in some way or another.
> making it easier to write cyclic graphs
I'd do it the same way I do it in any other language: initially create a DAG, then mutate it to create the cycles. This needs reference cells or something like that. Rust can do it already.
> reducing even further the cases where one has to explicity type lifetimes.
I don't see any way to do that in the vicinity of where Rust lies in the design space. What Rust has settled for (explicit lifetimes in type definitions, elision at use site) seems perfectly sensible to me. Of course, if a better alternative is found, that's a great thing.
> I'd rather see an axiomatic semantics for unsafe Rust
Luckily, this isn't an either-or! There are several groups of academics working with us on #2, while we work on #1. (among other things)
Good thing karma doesn't matter, and I have so much that a few comments being voted down doesn't affect me at all.
* What are some good ways to deal with a lot of C macros and C/C++ header files in general.
* What is a good build strategy for integrating with a big, say, cmake build system?
* How to choose APIs, other than always using the lowest common denominator (C) for everything? Put another way, how do you choose which parts to write in rust first so that you don't end up with rust code that looks like it's translated from C?
* How to work with other memory allocation strategies, like a region allocator (like postgres memory contexts)?
ConcurrentModificationException.
(0) RefCell can't be used in a multithreaded context. You have to use an actual Mutex.
(1) ConcurrentModificationException isn't reliable. It's thrown on a best-effort basis.
In any case, in a high-level language, just memory-safety is a very cheap guarantee. You also want to protect the internal invariants of your data structures - especially the concurrent ones!
Porting an inheritance model to Rust is ... interesting. You can emulate it with some macros, but usually you should redesign the idioms used based on the exact use.
At best people can hope that somebody comes up with a neat import system of c++ namespaces, a rock solid debugger and a fluid ide.
Rust will be like Go. Both pushed by corporations, and probably follow the fate of cobol, if history is to be asked.
Perhaps in adoption, but Rust has a huge feature over Go, which is that it can be integrated into existing codebases through FFI conventions, such that you can slowly incorporate Rust in safety critical areas (as an initial starting point).
Perhaps the author could get the book up to a presentable point before advertising it.
I think it is still wothwhile to present something you have instead of waiting once its done. Also the understanding "once its done" varies from the use case. This in your opinion unfinished book might be of use to others.
It is also good to have something written and share it with others to get feedback (as you did) at an early stage.
For that matter, the author might want to put his/her name on the book - I couldn't find anything in either the foreword or credits section.
Including how to port code from C or C++ to Rust, i.e., what it claims to be.
So far this is an OK overview of Rust's advantages over C and C++, but very very little of it is an actual "guide" to porting stuff.