Deciding to rewrite getaddrinfo in rust
blog.dkhenry.com
blog.dkhenry.com
Its getaddrinfo is considerably smaller at 90 lines, and there's no goto's, it looks comfortably readable at a glance. Maybe a nice blog post would be to compare both implementations to find out if there's a concrete difference in functionality and code quality.
edit: I should've looked at the 2nd Google result, it's a complete comparison of a bunch of libc's:
musl 1.1.5 was released on October 2014
glibc 2.19 was released on February 2014
Author of that page should probably add at the top of the page big header "2014". I can't find contact info to ask of him to do that though.
This is just a 10000 foot view of the two, but hopefully is helpful in starting to explain why musl's small size may not be all that advantageous if you want the kind of coverage a libc like glibc or freebsd libc has.
Software like glibc is battle-tested---it is _widely_ used on hundreds of thousands of systems around the world, and has been used (though not at today's massive scale) for decades. I understand that glibc is under active development and there is a lot of new code, but let's keep this in perspective:
Writing a new implementation of a system is a huge opportunity to introduce bugs. There is focus on these specific problems, but in the broader scheme of things, glibc is remarkably stable, performant, and feature-rich. A new implementation will have bugs, and those bugs might be less likely to be caught simply because the system will not be as widely used for quite a long time. Even formally proven systems don't address flaws in the actual program specification. (See "The Limits of Correctness" by Brian Cantwell Smith for a good discussion). Also relevant (which I'm reminded of in part because of his recent death): Peter Naur's Programming as Theory Building.
So even _new_ code to glibc has the benefit of a huge community of both developers are users to eyeball it and test it out in production on a huge number of systems.
So rewriting glibc may solve certain problems, but it's bound to create a whole lot more, considering the narrow range of issues that are being focused on. New Rust code will have undiscovered issues too, even if they're not memory or stack related. I feel that this effort might be better spent fixing and finding problems with glibc---and continuing the development of tools to find those problems, to benefit _all_ of our old C libraries and programs---than rewriting for the sake of rewriting.
Memory issues have a tendency to be the most severe types of issues from a security perspective. (Not always, but with a high frequency.) Other issues that may occur with a rewrite may be security problems as well, but the security track records of network-facing software written in old-timey C speak for themselves: memory safety issues make up a huge fraction of their vulnerabilities.
> I feel that this effort might be better spent fixing and finding problems with glibc---and continuing the development of tools to find those problems, to benefit _all_ of our old C libraries and programs---than rewriting for the sake of rewriting.
So we've been trying that for the past 35 years since C was released in 1978, and we've completely failed to move beyond making the same basic memory management errors again and again and again. I think that if we couldn't do it in 35 years, we aren't likely to be able to by just trying a little harder this one time.
As long as the public interfaces are still unsafe, it doesn't seem to me that there is significant latitude to qualitatively improve the security of a libc implementation.
Of course there is. The glibc bug we're talking about would not have happened if libresolv were memory safe.
To be more precise: I claim that, for a library for an unsafe interface, if the ratio of library interface area over the library implementation volume is high enough, it is not worth re-implementing in a safe language. I also claim that a libc implementation is generally well above this threshold.
I agree that this might not be the case for selected subsets of the library like libresolv.
The vast majority of the memory safety issues here are not at the public interface boundaries.
This seems a bit counter-intuitive given that serious problems are still being found years or even decades later. Perhaps battle-tested is insufficient.
And while new code is likely to have bugs, assuming it's already based on the original C code it would be less than if it was re-written from scratch based on a spec. New code can take advantage of the battle-tested nature of the original code even if it's in a different language.
Bugs will always exist. But that statement will apply equally to any software, regardless of language.
> New code can take advantage of the battle-tested nature of the original code even if it's in a different language.
That's why I referenced the Peter Naur paper on Programming as Theory Building---in practice, that may very well not be the case.
I'd be curious to see some research suggesting that the prevalence and severity of those bugs is identical regardless of development tool/language/runtime, but I'd be surprised if that's demonstrable.
The question at hand is not "will safer languages eliminate all bugs?" It's "will safer languages reduce the prevalence and severity of important classes of bugs?" I'd wager it's probably yes, but even if you disagree, I don't think that it's reasonable to suggest that because there will always be bugs we should never improve.
I intended to convey that software written in any language will have bugs, not that all languages will produce the same types of bugs.
My argument is based on the act of rewriting it---regardless of language. Many languages provide excellent guarantees, but that does not protect against bugs in the implementation itself (logic).
Despite being "battle tested", all of these C programs continue to have both memory and logic errors. I think a rewrite would have the same rate of new logic issues after the initial code review and testing. "bugs will always exist" -- sure, so if we have something that eliminates a class of bugs, why not use it? The other classes of bugs will be there (and probably in the same force) whether you rewrite or not.
A lot of these bugs get introduced due to cruft in old code as well. So there are a bunch of tradeoffs here.
This is the black and white security fallacy. Memory safety problems are, statistically, a huge quantity of security bugs. By eliminating them you drastically reduce the number of bugs.
(It's a close cousin to the "don't rewrite it" argument we've all heard via Spolsky et. al., even though they've ended up rewriting their shit several times over).
There is a fundamental shift forward in the types of guarantees you can make with the premises of a language like Rust vs. that of C that I have simply never heard one of these demagogues address.
1. Memory safety issues are the cause of a very large number of security vulnerabilities (often most of them for projects written in C or C++, depending on the software).
2. Memory safety-related issues have a relatively high probability of being turned into remote code execution, which is one of the most if not the most severe outcomes.
3. C and C++ projects have been empirically observed to have orders of magnitude more memory safety problems than projects written in other languages do.
4. The additional classes of security vulnerabilities that managed languages tend to foster do not have the combined prevalence and severity of memory safety problems.
So, we would be better served in security by moving to memory-safe languages.
1) Runtime overhead for some form of GC (D, Lisp, etc)
2) Rephrasing a program to satisfy a memory constraint checker (Rust)
3) Disciplined memory usage (i.e. Nasa C coding guidelines)
We don't have enough experience with 2 to indicate whether it will create new classes of bugs. We also don't understand the knock-on effect of managing memory differently - will functionally identical programs require more or fewer resources, more or fewer programmers hours, etc.
Rust may very well be the future, but we don't know for sure yet.
One thing we do know: options 1 and 3 have been available for years, but not widely utilized. What lessons can we learn from this fact to apply to Rust?
What classes of security bugs could possibly arise from Rust's ownership discipline?
Not all security bugs are related to memory. Many are related to improperly written algorithms (most crypto attacks), or improperly designed requirements (TLSv1).
Even Heartbleed was primarily due to a logic bug (trusting tainted data) instead of an outright memory ownership bug.
Does Rust automatically zero out newly allocated memory? Honest question, I don't know the answer.
Nobody is saying that Rust eliminates all security bugs. Just a huge number of the most common ones.
> Does Rust automatically zero out newly allocated memory? Honest question, I don't know the answer.
Yes.
This is a problem that will be there equally in all languages
Perhaps less so in languages with a better type system, but that doesn't affect Rust since there aren't any _systems_ languages with a better type system.
Oh, also: If you're implying that Rust's ownership discipline can create security bugs where there were none before, I consider that a real stretch. I'd need to see an actual bug, or at least a bug concept, that Rust's borrowing/ownership rules create before accepting this.
Note that this includes projects in languages that are themselves written in C or C++, which shows that there's some value in confining the unsafe code to a small and well-tested core library (in this case, the language runtime). Honestly, it seems like 50% of the value just comes from not using C strings, since pretty much every other language has its own string library that does not use null-termination.
The paper on the Limits of Correctness that I mentioned above does a good job at arguing my point. Even if you rewrote glibc in a language like Coq and formally proved its correctness, that doesn't mean that it's "correct" in the sense that its logic mimics glibc---there could be logic errors.
So you might gain confidence (or guarantees) in rewriting glibc in Rust, but in rewriting it you potentially introduce a host of new issues.
> Software like glibc is battle-tested
const char* s = ...;
while (*s++) { ... } void something(const char* const s);
Parts of it probably should be replaced with safer implementations, but as long as you're going for 100% compatibility you're still putting lipstick on a pig.
https://blogs.msdn.microsoft.com/vcblog/2014/06/10/the-great...
> fun aside, name and service are both const char* so I find it funny that they are set in the program, I do understand that from the callers perspective they don't change, but still bad form
… are you sure you know enough C to do this?
They just didn't want to force it into other OSes when ANSI C was defined and eventually POSIX took up that role.
(This is not to say that rewriting libc is a good way for people knowing Rust to spend their time. It's a ton of effort with the result most likely to be unused and if it is used, then it provides a small benefit to C and C++ programs at the expense of a rather scarce resource - Rust developers - having been diverted from benefiting Rust programs. If I were a Rust enthusiast, I'd much rather write libraries in Rust designed to be used in Rust programs so that more such programs get written more quickly than use Rust to put lipstick on the pigs that are C and C++.)
Once one understands it, getting rid of as much of it as possible makes a lot more sense. The replacement should ideally stand alone eventually, have stronger memory/type/integration safety, not assume what calls it (eg C code) will do things right, and integrate with C easily. Not sure how much Rust fits that as I lack details.
Result would've eliminated all kinds of crashes and host compromises due to C-specific errors. It is ironic that one of the best ways for a C application to be safe while using 3rd party code is avoiding C in that 3rd party code. :)
If it helps any, `getaddrinfo` is actually part of POSIX and not an essential part of C.
I say, "Go for it!" Kozlowski's attitude is exactly what we need. That's not saying it will ship in Ubuntu 18.04... It will be carefully vetted just like any piece of software, but it's got to start somewhere and I like that he's sharing his thought process with the community.
As people have pointed out, it may not actually be the Rust part of it that's valuable, but the organization and code quality. Rah rah!
> After reading about the newest glibc vulnerability, I have decided to see how much effort there is in rewriting parts of glibc in a safe language.
If I were a C programmer, especially if I were a C programmer who worked on glibc, I might feel a little bit insulted that remark. Safety is a complicated concept, and calling Rust "safe" and C "unsafe" oversimplifies things, especially if your Rust code includes unsafe blocks. See also http://www.tedunangst.com/flak/post/heartbleed-in-rust.
I don't think we have anything to lose by being as diplomatic as possible with these sorts of things. We can acknowledge that Rust's safety is a work in progress, and that we're still learning about the drawbacks.
This is the second post I have seen in the last couple of days of "glibc is bad lets rewrite it in Rust". However the problem is not that glibc is just bad but rather everything links to it. If other languages can't talk to Rust easy this would be a nicer problem to focus on first (before rewriting glibc).
I guess this is sort of how its done: http://siciarz.net/24-days-of-rust-calling-rust-from-other-l...
Rust -> C <- Other language. You have to create the C header files (I'm surprised Rust compiler or something similar doesn't automate this).
http://jakegoulding.com/rust-ffi-omnibus/
It's got a nice breakdown of how to bring rust libraries into a few languages. There are a few tutorials about how to make rust produce a static or shared library, but it's pretty straightforward.
The gist of it is: it's really easy.
https://github.com/Sean1708/rusty-cheddar
I haven't used it personally, but it looks useful.
> header files are a Rust-to-C++ issue only
Or... C? (And of course other languages too, as just discussed.)
Hence the "not necessarily" in my comment. The parent seemed to be painting this as extra unnecessary work; I was saying that a header file is _not_ necessary except for binding to C/C++.
If you want to write a C API that will be called from many languages; you should have a header file so that bindgen tools can consume it. If you want an API that will only be called by Python .... not so necessary, you may want a header file anyway but it depends on the situation.
> Or... C?
Sure, goes without saying :)
Rust in and of itself won't make things all better.
Correct me if I'm wrong (I haven't programmed in C++ in forever), but aren't "name" and "service" pointers to constant arrays of characters rather than constant pointers to (mutable) arrays of characters? In the first case, you're saying you have a variable that points to a memory location; the variable itself can be changed, but the data in memory at that location cannot. In the second case, you're saying that you cannot change the variable that contains the memory address; however, you can change the data in memory at that location.
(And if I remember correctly, I believe there are also such things as constant pointers to constant memory...)
const char *foo
const char * const foo
apt-get install cdecl
First, Rust allows unsafe code in the core language [not simply through FFI as in other safe languages]. So there's a gaping hole that we know will be abused (performance! performance!) right there. Even if we grant that this is not the case, here is the deal breaker:
The Rust approach to security depends on an ecosystem that follows the same approach. To think that this is more than an utopic dream is to enter cuckoo-land in my opinion. So we digress back to the castles-built-on-top-of-sand.
There's millions of lines of code written in unsafe languages plus all mainstream operating systems. The Rust approach will never work in this sort of environment.
An approach that might work however is unikernels and using a language that promotes design with failures in mind [and makes it extremely easy to rapidly rearchitect/rebuild/redeploy]. Erlang is the best example in my view, but there could be more.
Alas, Rust is not really suitable for this either due to its static nature.
Here is how to stop: (1) take out everything inflammatory ("It boggles the mind how utterly misguided", etc.) and make neutral statements instead; (2) take out the personal language ("you just don't get it", etc.); (3) replace vague grand claims with specific factual statements.
If you do this, you'll not only no longer be breaking the HN guidelines, you'll also be sharing what you know more effectively, which benefits all of us.
We detached this subthread from https://news.ycombinator.com/item?id=11162577 and marked it off-topic.
> we know will be abused (performance! performance!)
We'll have to agree to disagree about the rest. :)
Let me begin by introducing my premise: Rust will do NOTHING for security, it is not an improvement in any way, and at least as far as security is concerned, it will lead to at best colossal time waste, at worst actively damage the cause by misdirection, making ppl feel warm and fuzzy, providing illusion of security.
What??? B-b-but memory safety, I hear you say. How can this even be possible I hear you cry.
Memory safety (assuming that Rust completely solves it, which it absolutely DOES NOT) is but one (small) piece of the entire puzzle. In the same way that the skilled craftsman knows that he can't build a castle on top of sand, the skilled (and security conscious) software engineer should realize that memory corruption (and race conditions in the same unit/process) are not the foundation of the security chain. I dare say that a lot of the bugs exploited out there today (but not necessarily made public), are pure LOGIC bugs (that include race conditions) that stem from the interactions between extremely complex autonomous subsystems. Rust is simply operating at a too high-level abstraction layer to deal with any of these issues. This is the logical progression of Sergey Bratus's weird machines taken to the next level.
Some examples:
The Linux kernel. Any UNIX operating system taken as a whole. Any Windows taken as a whole. Any OSX system.
Anything built on top any of the above. Anything built on combinations of any of the above. Weird machines are not limited to isolated units, you know, they very well exhibit EMERGENT behavior.
Are you starting to get the picture now?
And then of course we still have the bugs that stem from unsafe languages, as long as there's a path that cuts through all the intermediate layers of "safe" code. Graphics drivers and webgl anyone? Truetype in the kernel?
Rust has plenty of things in its favor but to position it as a cure for issues that it doesn't even know how to address is a colossal mistake. We have given rise to a beast that threatens to consume us all. It's one thing to despair due to the immense complexity of the domain and quite another to step into an imaginary realm and solve illusionary problems that we create just to feel "safe".
Of course memory safety is one small piece of the entire puzzle. That's why Rust has features besides just being a memory-safe version of C.
You can have 2 components written in 100% safe language with no memory corruption and no race conditions.
You can have security critical race conditions that arise out of the interaction of same components.
How is Rust or whatever solving this? It doesn't.
If you give me a more concrete example, besides "you can have race conditions," I can give you a more concrete way Rust fixes it, besides "you can use types."
The race conditions I refer to DO NOT EXIST in the static pieces of code written in Rust or whatever ivory tower language. They are EMERGENT pieces of DYNAMIC behavior that only manifest at RUNTIME due to subsystem interactions.
And by subsystem I don't just mean pieces of code...
Rowhammer is basically mitigated if you don't let untrusted processes run CLFLUSH, right? That's sort of orthogonal to your choice of programming language (which might be your point, but then, I'm not sure I believe that Erlang or unikernels solves this either). It requires a world in which you've got something Native Client-like wrapping all userspace processes, which definitely seems like a good goal to work towards, and also seems easier to implement reliably in Rust than in C.
You're citing a hardware flaw as a reason not to pursue programming tools and techniques that produce fewer software defects? That's a false dichotomy; we don't forego researching cancer treatments just because some of the people that are saved will end up getting shot.
Earlier you wrote "Security is all or nothing".
That's another amazingly bad point of view. There is no "all." Nothing is absolutely secure. There is only raising the bar. Important things are provided more protection than less important things. That is the best that can ever be achieved. Since your premise relies on a fiction your conclusions about the value of Rust are meaningless.
I am trying to shift my thinking, but all I'm hearing is "No it can't." Can you explain why it can't?
For a language that certainly has the mindset, look at Erlang. Unikernels and Erlang will do more for actual security than a million monkeys cranking out perfectly safe Rust code.
"Decreasing an attacker's surface" is pragmatically, bullshit. No competent attacker (and that doesn't include just nation states these days) will be deterred by a decreased attack surface.
Also, proving a program "correct" (for some definition of correctness that presumably includes "secure") is undecidable, ergo there cannot be such a thing as a "100% secure language". No, not even Erlang, nor Haskell, nor anything which is remotely close to Turing-complete. So all we can do is, in fact, decrease the attack surface.
You often don't even need Turing completeness.
Actually, yes, it does. The borrow checker composes. Two correct Rust components for race conditions and memory safety will indeed compose together correctly. That's a great deal of the point of the whole thing, after all.
I don't think you've been keeping up with the state of the art in the last few years. Maybe less yelling and more listening is called for here.
Should engineers, admittedly lacking an absolute solution to traffic fatalities, forego seeking designs and materials that reduce the frequency and consequences of failure?
Rust delivers a higher degree of memory safety. It is a better tool in the tool box and arguing that engineers should ignore it because their efforts are inherently flawed won't work, any more than arguing that a new headlamp or traffic light design is futile.
Also, you should understand now what you're up against. The legacy stack is riddled with flaws; every day we face a deluge of security notices, some large fraction of which are caused by memory safety problems. In other words, your opponents in this debate have an endless supply of ammunition.
Good luck with that.
> nobody has a solution to fix the problem today.
You're entire premise revolves around 0.01% of the population. That's not a good basis.
There's about 34,000 automobile related deaths annually in the US[1]. There's about 323,000,000 people in the US.[2]
The point I'm making, is let's not get carried away with exaggerating how "bad" a "problem" is. According to the CDC, more people die annually from accidental poisoning (about 39,000) than automobile related accidents - but I don't hear all the calls to "solve" that problem.
Will things be "safer" written in Rust? Maybe. Does Rust solve everything? Absolutely not. Yes, a lot of bugs are "unsafe" memory access issues, but a lot of bugs are not. Re-writing everything in a pet language that happens to be popular today in 2016 is not a good call. Rust is fairly new, and nobody has any idea if it will stand the test of time. C has, clearly.
Instead of getting out our pitchforks, we should rally to guarantee these critical projects are fully funded and staffed with full time developers. That alone, will yield far greater results than some lofty goal of completely re-writing 100's of thousands of lines of code which have been working, tested, and matured for almost 3 decades.
I think that's pretty good evidence of the grandparent's point: no, you're never going to make anything completely safe, but society can and does put a lot of effort toward harm reduction.
Mine is the basis of billions of dollars of engineering and untold amounts of legislative and regulatory efforts around the planet. A lot of people seem to think it's a pretty important basis.
let's not get carried away with exaggerating how "bad" a "problem" is
On one hand you ask others not to exaggerate, next you're talking about pitchforks. The story we're discussing is about someone reimplementing one little 300 line procedure. I made no more of Rust than to call it a better tool in the toolbox. Calm yourself; these pitchfork wielding exaggerators you speak of are fictions inside your head.
stand the test of time. C has, clearly
The buffer overflows, dangling pointers, double frees, use after frees and leaks recorded daily as CVEs is the most demonstrative thing time has revealed about C et al. If C had past the test of time Rust wouldn't exist.
A huge number are. For example: https://www.cvedetails.com/vulnerability-list/vendor_id-72/p...
> That alone, will yield far greater results than some lofty goal of completely re-writing 100's of thousands of lines of code which have been working, tested, and matured for almost 3 decades.
Windows, the popular Web browser engines, and antivirus software are all examples of systems software that are very well funded. But they still regularly fall to memory safety problems. The converse is also interesting to look at: djbdns and qmail are examples of poorly funded software, but they have had very few memory safety problems.
I agree with the general principle that funding helps security overall, though.
I see you've used it. I think it sets a bad tone for an argument.
Visit https://www.cvedetails.com/vulnerability-list/vendor_id-72/p.... Count the number of memory safety issues. The first five, for example, are all memory safety problems.
http://regex.info/blog/2006-09-15/247