I am curious why they were advised not to use Go. Probably not a safety concern.
Edit: cgo != Go. Thanks for the responses. I have done a bit of Go, but just pure Go.
Edit: cgo != Go. Thanks for the responses. I have done a bit of Go, but just pure Go.
A pure-Go rewrite might be an option (in fact Tor seems pretty firmly in Go's use cases), but that's not what the Tor team is trying to do.
[0] https://dave.cheney.net/2016/01/18/cgo-is-not-go
[1] a cgo->c call is ~100 times more expensive than a go->go call, and ~400 times more expensive than a c->c or rust->c call https://www.reddit.com/r/golang/comments/3oztwi/from_python_...
[2] https://www.cockroachlabs.com/blog/the-cost-and-complexity-o...
But yes, Go pointers in C code is bad juju.
Sorry, my reply was too brief. I wanted to add that the ergonomics are bad, not just because of freeing memory (for which the inconvenience can indeed be reduced with finalizers and/or Close() methods plus defer). But rules such as this one make Go->C FFI unergonomic as well. To give one example: many linear algebra libraries (e.g. Tensorflow) have their own wrappers around raw arrays to represent tensors (with their dimensionality) [1]. As a consequence of this rule, one cannot just a pointer to the first slice element to such functions (since a pointer to a Go object would be stored in a C struct), but have to malloc an array and copy over data from the slice to the C array.
[1] There are other issues, such aligning slice memory to 16-byte boundaries.
(Of course, I think the change led to a better overall API. Go just helped me get there in a circuitous way.)
It turns out (or so I hear) that Google statically links everything in production, and has been using C++ as a language to implement HTTP endpoints for a long time. So Go is a better C++ for what they want out of a better C++; for the rest of us, it looks more like a compiled language along the lines of Python/Ruby/etc. with a nice deployment story. If you want that out of your better C++, Go is great. If you want to reimplement all of Tor from scratch, Go certainly seems like a reasonable choice.
But as a result of these priorities, Go basically doesn't have interoperability with the platform ABI as a goal. (For some combination of historical reasons and the lack of complicated features in C, the platform ABI on just about every platform these days is a C ABI.) Rust does; it uses a standard compiler toolchain (LLVM) instead of what's basically a custom one (Plan 9), and the standard toolchain knows how to generate calls that follow the C ABI. Rust doesn't have a runtime of its own, and it's safe to directly call into a Rust program from some arbitrary point in a C program. Rust's allocator doesn't care if you do stupid things with pointers it allocates, as long as you give them back eventually. Rust doesn't create threads on its own unless you ask. Rust functions use the normal stack. Rust on UNIX uses the platform libc. And so forth.
It's possible to call C code from Go and vice versa, just as it's possible to call C code from Python and vice versa. But Go is not best tool for this particular job.
Rust has some interesting features, but being C isn't one of them.
What would be the advantage, just improved memory safety guarantees for people working on the project?
If that's the case start again from scratch and the first thing you do is build a handle based memory management system that everything must go through Safexxxx() versions of everything and then explicitly enforce no other memory access patterns.
If you want to do concurrency, build simple threads with input queues that would just be like channels in Go.
You can use a completely functional actor model in C. Is all this object oriented dangling pointer stuff that is scary, but there really isn't any need for that.
Software pipelines with slab allocation or ring buffers virtually guarantee no leaks and are very scalable and cache friendly.
That would be my personal recommendation.
Rust provides, among other things, a decent type system that can encode some of your invariants in a manner that's a lot clearer - and every operation goes through the equivalent of your Safexxxx() functions by default, making the actual logic of what you're trying to do further clearer. A reasonable error handling mechanism is also built into the standard library and partially into the language, rather than checking return values for -1, or is it 0, and is the actual error in errno or was that function actually meant to return a value between -1 and -100 corresponding to the error, etc etc.
The advantage of Rust is, honestly, that it has a community of people who are excited to do this sort of work in a language. At least 50% of the advantage of any particular language is not the benefits of the language directly, but the community around it (that derives indirectly from the benefits of the language, but also from marketing and other things). Some time ago I needed to write bindings to SCM_RIGHTS / CMSG_*, which is probably the trickiest part of the libc API (all of cmsg(3) is macros that do stupid things with casts). It was annoying, but it led to a better interface than C itself offers, and someone else found a bug in the OS X handling. I couldn't expect that if I were working on my own custom reimplementation of libc.
And is funny you talk about community, since as the guys on slashdot and others pointed out, the community of people who can use and work in C effectively is orders of magnitudes more than Rust. Have you any idea how many Linux Kernel developers there are alone?
http://m.slashdot.org/story/324469
Building API's specific to the use case is part of our jobs as professional developers.
Abstracting a few things things isn't "rewriting libc", that is just general practice for most decent size projects.
Anyway, the decision is made, so the whole thing is moot at this point.
The key word here is "effectively." For the purposes at hand, effectiveness includes memory-safety. Do you know of a single project that implements cmsg(3) in C or C++ in a memory-safe, well-typed, cross-platform way? Or a project where I can submit a pull request and expect it to be reviewed, tested across platforms, and fixed?
I do genuinely believe that the community of people who can use and work in C effectively, in the sense of effectiveness that I and the Tor Project are interested in, is orders of magnitudes smaller than Rust.
I have a very good idea of how many Linux kernel developers there are - and also how many high-severity security bugs there are. I'm a coauthor of a research paper where we wanted to talk about exploitable security bugs in Linux, so we sat down and found a local privilege escalation in hours (CVE-2009-0024).
+1 for the rest of the explanation.
Therefore a slow transition of rewriting parts of the code in a safer language and having the core still in C is much less feasible with Go. With Rust you can easier just compile some object files and link them into your application.