Pre-Pooping Your Pants with Rust | Hacker News Reader

Pre-Pooping Your Pants with Rust

cglab.ca

129 pointsManishearth97 comments

I hope that the Rust devs end up doing the "right thing", which from this article seems like going back and fixing Rc, instead of just marking mem::forget as safe. They're still pre-1.0, and anything they do now they will be stuck with for a long, long time, especially if Rust succeeds as much as I hope it does.

Delaying 1.0 by a few weeks may seem like a big deal, but ultimately it is a self-imposed deadline. It's great to have those, but following them dogmatically might not be the best strategy. In cases like this, I generally lean towards slowing down and doing things right: otherwise you will pay the price ten times over later.

(That said, while I understand this issue, I don't know very much about the context in the Rust community, so I'm not actually sure that mem::forget should be unsafe. It was just the impression from the article and from previous Rust code I've read/written.)

There isn't an acceptable way to fix Rc. It wouldn't be hard to require it not to have cycles, but cyclic data is actually a major usecase for Rc. A cycle collector wouldn't fix any of the unsafety here unless it was run on literally every drop, which would be far too slow to be acceptable in a systems language. An opt-in trait to prevent leaking specific types would be practical, but the implementation details aren't settled and it can be done backwards-compatibly, so there's a strong feeling that it shouldn't block the release. Any other attempts to determine whether cyclic data structures always eventually drop their contents enter very-active-research territory and would be more like a ten year than a few week thing.

Precisely. Rc is not easy to "fix": the fact that it allows cycles is part of its design.

There's a reason why finalizers aren't a widely used technique in Java and why Python's cycle detector simply stacks things with destructors in an out-of-the-way location.

Those languages have bigger problems than Rust here. Rust can still recover correct behavior for APIs like scoped because it can ensure (using lifetimes) that the relevant value never escapes the stack frame. In the languages you mention (and most other languages with a "with" or "IDisposable" equivalent), the "close" equivalent, or even the finalizer, can be called on objects that still have valid references to them hidden in another object or thread.

I was actually speaking of the original finalize method it Java[1]. There's no guarantee that it will be called, there's no guarantee when it will be called if it is, and as a result it shouldn't be used to reclaim resources or much of anything, really. Where "should" is so strong in this case that I've never seen it in the wild, even given the N-monkeys code that I have seen.

[1] http://www.informit.com/articles/article.aspx?p=1216151&seqN...

I don't know enough about how Rc is used to know if this is an acceptable possibility, I just wanted to mention it in case it is possibly helpful.

I developed a reference-counting system in my library "upb" that handles cycles and never leaks objects. It is precise for no-longer-mutable objects (objects which may no longer change the set of other refcounted objects pointed to) and less precise for mutable objects.

The basic idea is to reference-count groups of objects instead of single objects, and define the groups such that no reference cycle spans groups.

If you introduce a link between two refcounted objects A -> B, then A and B's groups are merged. Now refs/unrefs of A or B ref/unref the (merged) group. Nothing in the group is freed until both A and B's refcounts both fall to zero. This conservative group-merging whenever you create a link ensures that no reference cycle spans groups.

This is imprecise and relatively wasteful if the group grows really large. But you can make it totally precise for any subgraph of refcounted objects that you're willing to freeze. At freeze time, compute strongly-connected components, and each SCC becomes a refcounted group. For any frozen subgraphs, the refcounting is totally precise. And unfrozen objects can reference frozen ones (but not the other way around).

If your application ends up freezing most of the graph, this works great. Or if you're ok with the collection being pretty coarse for your groups of objects, it also works great. If you have an application that can't freeze the graph and wants pretty precise collection, this doesn't work so well.

More info about my scheme is in comments in these headers:

https://github.com/haberman/upb/blob/master/upb/refcounted.h

https://github.com/haberman/upb/blob/master/upb/refcounted.c

I always was curious if the semantics of this scheme would play nicely with the ownership system in Rust.

btw, I never did this in my library because I didn't need to, but you could also support an operation that computes SCC's to minimize the groups without freezing the graph.

The minimized groups would still merge with other groups if you created links between them. But this would be a way to make the refcounting more precise without strictly requiring that subgraphs become immutable.

You'd have to delay 1.0 by more than a few weeks -- making mem::forget unsafe means marking all memory leaks as unsafe, which is unsolvable.

I'd prefer the `Leak` based solution too, but it's going to cause a _lot_ of churn. 1.0 has already been delayed often, every time it gets planned and then not followed through on because the date is fuzzy and ignorable. This time they've set a concrete no-nonsense date which we should follow through on IMO. We already had a second alpha.

I guess it's better that way than saying something broken is "ok" only because "I promised myself that I'll finish until next Monday". It's more like Ubisoft-style, or something, except deadlines really matter for them, because money depend on it, and they cannot just "move the deadline" because of all the advertising must be timed, people tend to buy games and go to cinema more on specific dates, etc.

For project like Rust deadline doesn't actually matter that much, grasping for it is almost stupid. The only thing that actually can suffer from breaking it is self-esteem, which should't worry a reasonable man much. But magic "1.0" number does matter a bit more than just deadline, because after it there's no "breaking changes".

So I'd be much happier if Rust wouldn't reach 1.0 for the next 2 years, but would actually become satisfying instead. Somehow "non-stable but working" is better suited for making software than "broken and stable". We have plenty of "broken and stable" out there already.

Rust is not seeking pure programming perfection, it is seeking to be useful and to fulfill its goals of safe systems programming, and it is fulfilling these goals with flying colors. There's absolutely no reason for this to push the release date. There was a bug in a stdlib API, and that issue was fixed weeks ago. Trying to pivot the language by taking a fundamental feature back to the drawing board for nebulous benefit would be utter foolishness at this point.

Please see my comment below (https://news.ycombinator.com/item?id=9447938). If this were a problem for Rust's basic memory safety guarantees, we'd certainly consider delaying release. There are also ways to pursue the `Leak` approach in the future backwards-compatibly.

It's very disappointing that the Rust community has opted to go back and change the definition of "safety" instead of actually tackling this problem head-on, all due to a self-imposed deadline there is no need to actually meet.

The definition of 'unsafe' has not changed.

Rust's basic guarantee has been, and continues to be: unless code uses `unsafe`, it is guaranteeed to be memory safe. The issue under discussion doesn't change that -- it's largely about what such `unsafe` code is allowed to do and assume. I've laid this out in a bit more detail in a comment below (https://news.ycombinator.com/item?id=9447938)

Please see my comment below (https://news.ycombinator.com/item?id=9447938). There is no risk to Rust's basic safety guarantees here, and there are several ways forward (including introducing things like `Leak` backwards-compatibly later on).

There is a general problem with combining destructors and automatic collection of reference cycles, and its one that several languages have unexpectedly found themselves facing.

For example, Java has the function System.runFinalizersOnExit, but if you look at the documentation you'll see that it now says, "Deprecated. This method is inherently unsafe. It may result in finalizers being called on live objects while other threads are concurrently manipulating those objects, resulting in erratic behavior or deadlock." http://docs.oracle.com/javase/7/docs/api/java/lang/Runtime.h...

The problem is discussed in detail in Hans Boehm's 2002 technical report “Destructors, Finalizers, and Synchronization”. http://www.hpl.hp.com/techreports/2002/HPL-2002-335.pdf

Boehm's key points are: (1) When a destructor on an object O runs normally, anything that O points to is still alive (because of the reference from O), but when a cycle is collected, not everyone can go first: that is, all but one destructors in the cycle have to run after one or more of their references are dead. It is hard to write destructor code that is safe in all cases. (2) Any destructor that needs to update a concurrently accessed data structure has to take a lock, but destructors on cycles run asynchronously with respect to the rest of the program and so an unlucky timing leads to deadlock.

If this problem was better appreciated then language designers wouldn't go down the rabbit hole of trying to figure out how to make destructors work together with automatic collection of reference cycles, and instead try the alternative approach of providing mechanisms for the program to run destructors synchronously.

In the Memory Pool System we use a message-passing interface: http://www.ravenbrook.com/project/mps/master/manual/html/top...

Call Scott Meyers.

So you cannot safely assume your destructors will be called, and you need to proactively take steps like this in case they aren't. Nothing in the natural use of the language cues you to do this, so you need a moral guide like "Effective C++" for Rust.

This was the kind of accidental complexity that Rust had hoped to avoid. It seems like Rust has avoided a lot of it. But I guess it was entirely too optimistic to hope Rust could avoid all of it.

Note that this is only something you really need to think about when writing `unsafe` code, which basically unlocks a whole new set of rules that you need to worry about (zero-sized types in offsets, manual allocations, safety boundaries, GEP/aliasing rules, uninit memory, double destructors, etc). This is a drop in the bucket that anyone venturing to use `unsafe` will have to face in some kind of "guide to unsafety".

The problem is that all of this is safe code. The problem, to my mind, is that you can write mem::forget (as in the code here's safe_forget)* without ever using the unsafe keyword*.

Edit: I was wrong. The problem isn't with forget. This is hairy.

The 'mem::forget can be written in safe code' meme comes from https://github.com/rust-lang/rust/issues/24456

Which, as you can see, uses Rc, which uses unsafe code in a way that leads to the soundness bug. So that's not strictly true, or rather, doesn't really change anything, as it relies on the same bug.

nikomatsakis, in that thread:

"1. There were other ways to forget even before Rc, at least in some cases. For example, if T:Send holds, you could send the value to a thread that runs an infinite loop, or which is deadlocked on a port.

"2. It is true that one cannot assume that a destructor will run, and hence that forget is not itself unsafe (rather, it is unsafe to write a dtor that must run)...."

And alexcrichton:

"I commented on #24292, but the gist is that there are multiple ways to leak memory today (e.g. #14875 and #16135), so a targeted solution at Rc may not cover all use cases. Although as I mention in #24292 these other bugs can also be considered separate bugs on their own which need to be fixed regardless (but sometimes is quite difficult to do so)."

I especially liked dgrunwald's comment:

"A safe mem::forget has the advantage that it makes it easier to write the counterexample proving thread::JoinGuard unsafe. Safe mem::forget makes it more likely that people will know that destructors are not realiable, so they can avoid repeating this mistake."

But then, "Put a big freaking spike in the middle of the steering wheel and get rid of the airbags and seat belts" has always appealed to me as an automotive safety approach.

Yeah, I mean, I guess that leaks in general are possible, which would still let you write `forget`. You're right about that.

But the unsoundness RC bug still relies on unsafe code which was written incorrectly.

I think you need to split the unsoundness bug, which happens because thread::scoped uses unsafe code, from the "you can write mem::forget in safe code" bug, which is arguably not a bug but is at least an enormous footgun waiting to happen.

When the footgun goes off in unsafe code in the standard library, you get use-after-free and memory unsoundness. When the footgun goes off in safe code written by mere humans, you leak arbitrary resources.

The fact that you can write mem::forget in safe code is nothing more than an argument that mem::forget itself should not be marked as unsafe.

The fact remains that as long as you are not using unsafe code yourself, you don't really have to worry about anything. The category of destructors that leave the world in an unsafe state if not run is AFAIK a strict subset of the category of destructors that require unsafe code to be written.

The problem here looks like it is with Rc, which does require unsafe code. I haven't checked, but I don't think the destructors in the thread::scoped issue are unsafe. Even if they are, I can easily imagine another situation where a programmer assuming a safe destructor is always called causes a bug, which is exactly the problem with thread::scoped.

And Rc only looks like it is the problem. Rc is doing exactly what it says it does, as safely as it can. The fact that you can use it to avoid having a destructor called is a side-effect of it doing what it's supposed to do.

If main is safe, mem::forget is safe (remember, Rc is just playing the part of mem::forget), and the destructor for JoinGuard is safe (which it is; I just checked), where is the unsafe code?

`unsafe` is a stateful property, not a local one. The specific invariant here is that you cannot rely on destructors for memory safety, which the old thread::scoped API was doing. The actual unsafe code is actually here: https://github.com/rust-lang/rust/blob/master/src/libstd/thr...

(I myself have been guilty of this same error, of assuming that `unsafe` regardless of its position cannot have far-reaching effects on the overall state of your code.)

The important thing is this: using safe Rust, and using any safe stdlib API, memory unsafety is impossible. This includes stdlib APIs that use `unsafe` internally, such as `Rc`. Proving those interfaces safe is the burden of the Rust developers, and if you can show that there has been an error in these proofs then it is a drop-everything sky-is-falling defcon-1 situation that they must address, up to and including unreleasing the language if no suitable solution came to mind (fortunately, this is quite unlikely).

> where is the unsafe code?

In the thread spawning. The destructor itself isn't unsafe, but it was being used as a guard to enforce the safety of the threading code.

> I can easily imagine another situation where a programmer assuming a safe destructor is always called causes a bug, which is exactly the problem with thread::scoped.

There's a difference between "a bug" and "memory unsafety". If the bug is that the value the destructor guards isn't released (for example, if a File is leaked, the associated file descriptor will never be flushed and closed), then that's not Rust's problem. The programmer probably wanted that file descriptor to be closed, but there's no memory safety violations going on there, and in fact nothing worse than simply stuffing that File into a global array/hashtable and forgetting about it.

The problem with thread::scoped is actually fairly unique. The destructor is not doing much besides just calling join() on the forked thread. The problem is that the JoinGuard object doesn't actually control any access to anything at all, it exists purely as a proxy to represent any values (particularly stack values) that are borrowed by the forked thread. Because it's only a proxy, leaking the guard does not actually make the guarded values inaccessible.

I can't think of any situation besides multithreading that has this same problem. Every other situation involving a RAII object with a lifetime being leaked (and therefore outliving its lifetime, which is technically a violation of borrowck) by definition makes the object inaccessible, which means the data it's protecting cannot be reached. For example, a struct that contains a &-ref to some value may outlive the referenced value if the struct is leaked, but this is still memory-safe because the reference will never be dereferenced. If the reference can be dereferenced, that means code exists that has visibility on the struct, which means the struct hasn't actually been leaked yet, and as long as it hasn't been leaked it's still safe (e.g. because borrowck will still enforce the lifetimes).

Multithreading is special because the forked thread is what's actually accessing the values, without ever going through the RAII object. This means that borrowck cannot guarantee that the RAII object is still alive (and therefore that the "protected" values are still valid). This was considered to be ok as the RAII object would join the thread (therefore ensuring it had finished executing) before destructing, but the ability to leak objects means the join may not happen.

Ultimately, this means that JoinGuard is the moral equivalent of a Mutex that coexists with its protected data rather than owning it (thereby making it possible to write code that accesses the data without holding the lock). It's a bit better than that, because you have to actually go to some effort to leak the JoinGuard as opposed to just doing it accidentally, but the point is that borrowck cannot guarantee that the code that accesses the data does not outlive the guard.

And all of this comes back to my original point, which is that I can't think of another situation where it's actually unsafe to leak the RAII guard (as opposed to merely being a bug) without there being `unsafe` code in the guard itself. Note that the example given for PPYP, Vec::drain_range, uses `unsafe` code in its destructor.

A quick observation from an outsider: The article goes into a lot of depth, seems highly technical, well crafted and researched but the whole "pre-poop your pants" thing just ruins all that and makes it looks childish. I understand that not everything about development has to be serious, but I also think that aligning form and content is the surest way to make sure your message comes across.

I guess it's a personal thing. I don't know much about Rust, but the title explained the severity of the problem pretty good right away.

I'm hoping this design pattern name catch on. PPYP design pattern.

Sometime programming can be boring and then you come across PPYP and it makes it fun again.

First it was dry and boring, and then someone mentioned "poop" and then the same subject was hilarious! I don't get it...

First Ruby, then JS, now Rust.

What is your point?

I guess the parent means something that has to do with the community being a bit hipster / infantile? Only guessing though.

The entire infantile humour / brogrammer / hipster / drama phenomenon. It started in Ruby (_why, Zed Shaw, GitHub, etc) then moved into JS as Node and CoffeeScript became popular with Rails users, now it's in Rust (again via Ruby).

I don't think it's always the same people or that it has anything to do with Ruby itself, it just seems like there's a certain culture that first grew in the Ruby / Rails community.

FWIW I actually thought the title detracted from the otherwise excellent contents of the post.

This post is the first infantile thing in this vein I can remember seeing related to Rust, and, as far as I know, the author has no connection to the Ruby community you describe, they're just choosing poor titles of their own accord (I agree with Steve that the title detracts).

I'll reuse my reply on /r/rust to defend the naming:

> I like the naming and feeling of "Pre-Pooping Your Pants" pattern. We are doing the intentional leak only to be (hopefully) collected later, which may not happen but it still remains memory-safe even in that case. Still this is not very desirable and the name clearly indicates so.

A term for code written in an undesirable way to tie you over in the short-term is a "hack".

Not all hacks are bad. But, if I see "// this is a hack to..." or "# this is a hack to..." or similar in code, that is a signal to me that the developer (1) either didn't have time to perform the right thing but could have with more time (within their lifetime), or (2) could not feasibly perform the right thing under constraints given (within their lifetime).

Over the past ~8 years, "hack"'s use has switched back to a much more positive sense- possibly because of HN, or possibly because people read Eric Raymond http://www.catb.org/esr/faqs/hacker-howto.html, but "hack" != "hacker". The negative sense is still used and understood by most developers, and it comes from two things: "HACK... HACK... HACK...", like the sound of an ax attempting to chop down a tree (brute force), and the other definition of a hack not used as much these days, which is "work for hire especially with mediocre professional standards".

I think we should at least consider PPYP to be a subclass of "hack", or perhaps just use the word "hack" instead of PPYP.

kludge is a better term for an undesirable short-term fix. hack implies clever or elegant.

I couldn't agree more. the article is well written, but the title puts me off a lot.

How am I going to show this to my boss, and have him take it seriously with such an infantile title?

Why would you want to show it to your boss?

I tend to keep him updated with rust news, so that when the day comes, I've laid enough groundwork to be able to do something in it without having to explain what it is and why etc.

And part of that is showing flaws, and how they're resolved/worked around and so on.

You have the right to your own feelings, and I agree that the title may be off putting. The audience for the article is rust core contributors and rust fans, and they're a very informal bunch. If rusters are like similar communities I know, talking about poop makes a blog post more likely to spread.

EDIT: I do not want to argue with Dewey2, so I'm putting this here: the women in the rust community that feel uncomfortable when they are addressed as "guys" have every right to feel that way, and I commend the rust community on making that stand.

There's nothing to suggest all other rust devs like this way of presenting it. I don't. I agree it's off-putting even if the issue itself is interesting.

> How am I going to show this to my boss, and have him take it seriously with such an infantile title?

Probably the same way you would explain to your boss what KISS principle means or what WTF-metric for source code is :)

I don't know the wtf metric, but whatever it is it's not about "poop". So I'm going to say it's different.

It's an old joke, but it has a bit of wisdom.

Illustration: https://davidlongstreet.files.wordpress.com/2009/04/wtfm.jpg...

I think Pre-Poop Your Pants is a really clever way to package the core idea. I agree that the article is well crafted, and I don't think it's childish to talk about poop in this context.

It's not an "official" article, just a blog post by a contributor. Informal is okay IMO.

Why on Earth are we still using such unsafe languages?

If the author had written this in a modern, safe language like Rust this would not have happened.

Funny.

But the problem here is complex and hard to solve. Cycles and memory leaks are nigh impossible to solve completely without sacrificing on that altar ability to create custom data structures or no-GC by default. Which was non-starter for Rust.

Well, I'm still not quite proficient with Rust, so maybe my feelings are misguided, but the whole situation seems really terrifying to me. Like, way too terrifying than that we could just make a point about "not doing like that again" and calmly move on to the 1.0.

After all, why Rust? Because we're all tired of problems with C++ and stuff. And the point isn't really about making a language that would be safe "as often as possible", it would be quite terrible goal, actually. The point is, I believe, to make it nearly impossible to write unsafe code accidentally, without even noticing it.

And again, maybe it's just me, but what I understood from that post is totally non-intuitive to me. I don't really understand what are the guidelines and how many more gotchas like that there could be. "Not to mess with unsafe code" is not really good guideline, because it turns out that it can be not quite obvious that the code is unsafe.

Well, the thing is, until Rust is formally tested, as in a subset of Rust is proven in CoQ to be safe, you don't have any guarantees.

I assume much of it is safe, with few probably hidden caveats, hidden in more complex logic. In C++ writing code that causes this behavior is trivial, here you need to jump through a lot of hoops. So it offers you comparatively more safety, but this kind of bugs are potential gold mine for crackers.

Anyway core developers are on the scene deciding how to tackle this issue.

Oh, sure, but let's be reasonable. If it's something like the notorious sort problem in Java, I'd say it's ok. It's preferable when things are simple enough that somebody with a good understanding of the whole system can make assumptions about its safety without using CoQ. If things are complicated enough that we actually need Agda/CoQ/Idris to feel safe — seems that things went really wrong at some point.

> but this kind of bugs are potential gold mine for crackers

Exactly! And I'm just saying that a problem which is "not very likely to trigger, but is very hard to find if happens" is far, far more dangerous than a problem which is "easy for a novice to miss, but every experienced developer would notice".

      > If things are complicated enough that we actually need Agda/CoQ/Idris to 
     feel safe — seems that things went really wrong at some point.

I disagree. Complex software/hardware is by its nature full of bugs. What Rust set out to do is complex and it's bound to have bugs, but doesn't mean it's reason to abandon it, no more than we should discard LHC or ITER because they had issues starting up.

Whether it's a wrong invariant on TimSort or a really obscure multi-threading case, or a weird glitch in time library, its still an error. Each one is exploitable.

ALL software could use some coverage/tests and theorem proving. I just wish Rust started with a proven safe subset.

      > Exactly! And I'm just saying that a problem which is "not very likely to 
     trigger, but is very hard to find if happens" is far,   
     far more dangerous than a problem which is "easy for a 
     novice to miss, but every experienced developer would notice".

Again, I disagree. The matter of security is one about making the assailant less likely to breach your software. If you take two comparable pieces of code, the one with more holes will take less effort to break.

If your language offers fewer avenues of exploit, makes these exploits more expensive. A rare bug is worth more than a run of the mill bug. Which prohibits the number of assailants which could purchase it, which reduces number of potential attackers.

Please see my comment below (https://news.ycombinator.com/item?id=9447938) -- a key point is that, as always, you have to write `unsafe` to get memory unsafety in Rust, period. This is Rust's core guarantee. The issue being reported here is a safety problem caused by a particular use of `unsafe` blocks. The debate is largely about what unsafe blocks should or shouldn't be able to assume.

We need a Rust for unsafe code. Which is to say, goalposts are moving! That's exciting.

Boring edition for people who have feels about maturity: http://cglab.ca/~abeinges/blah/everyone-peaches/

( people getting mad about PPYP is really a victory, though; the second last bullet was the most important:

> It's vaguely incoherent and meaningless on its own: this is a property inherited from its connection to the venerable Resource Acquisition Is Initialization (RAII) pattern.

I'm mad about RAII and not going to take it anymore! )

Putting stuff in a HashMap that you'll never ask for again is a kind of leak. Allocating stuff on the stack and then looping forever is a kind of leak. The case where you put it on the heap and forget to ever free it is a specific kind of leak that people seem particularly terrified of for whatever reason.

I too am particularly terrified of this kind of leak. The reason is that it is the only kind of leak mentioned that is not clearly a programmer mistake.

C++ has a whole class of "mistakes" that one cannot be aware of simply by reading the language specification. Common ones such as writing a constructor and destructor but not a copy-constructor are now compiler warnings, but many will pass under the radar. As a result, to write correct code one needs to be aware of multiple conventions and rules and patterns that are external to the language. When reviewing code one needs to do more than observe that it compiles and accomplishes its stated task, there are probably several organizational standards that need to be satisfied as well to be safe.

This is precisely the kind of thing that Rust was supposed to solve. The goal was to have a language that didn't require an "Effective <xyz>" book to be memorized to do code review. A language where any rule like "You should always free resources in advance in case the destructor doesn't run" is captured statically by a compiler error.

So yes, I am particularly put off by this error.

Well, it's worth noting that this error happened because unsafe library code relied on a property that wasn't actually true. The language itself is working just fine, Rust cannot and will not prevent all bugs.

Scrapping the guarantee that destructors will be called is tantamount to scrapping RAII. thread::scoped made the assumption that destructors were called before a borrowed value was returned, and it led to a memory-safety error. If you don't use unsafe, you won't get memory-safety errors, but you can get plenty of other nastiness.

The guarantee that destructors are called when values leave scope is fantastic. It means that one can leverage memory-safety into arbitrary-resource-acquisition-safety. It violates the principle of least astonishment in a brutal way if Rust's borrow checker can statically guarantee that a value no longer exists, but can't know whether or not its destructor has been called.

> A language where any rule like "You should always free resources in advance in case the destructor doesn't run" is captured statically by a compiler error.

This only shows up if you're specifically opting into unsafe code by using the "unsafe" keyword. Rust doesn't (yet) try to ensure safety of unsafe code, because it's unsafe. As long as you don't step into unsafe code, you can ignore this entire article.

Which unsafe code is that? In the article, I don't see any use of "unsafe" in safe_forget[1] or either main[2] in the first part of the article.

[1] Which creates a reference-counted cycle containing the thing-to-be-forgotten, ensuring the finalizer is never called.

[2] The first main simply demonstrates that the destructor is never called, which is ok in itself but causes problems when you're doing smart things in the finalizers, as in the second main, where the code uses the destructors to ensure the threads are terminated and the programmer is assuming the borrow checker is going to warn them if the threads aren't terminated.

https://github.com/rust-lang/rust/blob/master/src/liballoc/r...

Making mem::forget safe makes that point moot.

https://github.com/rust-lang/rfcs/pull/1066

Forget isn't the reason that that block is unsafe.

Right. That block is unsafe because it's doing unsafe things. (ptr::read and alloc::heap::deallocate)

But the use of Rc is kind of a red herring; if mem::forget is marked safe, then you don't need safe_forget because you can forget things (fail to execute their destructors, specifically) safely anyway.

The problem is that destructors aren't guaranteed; the bugs in the thready thing (and potentially unbounded other things) are symptoms. Drop needs big, red warning signs.

The power of a notation, and a type system, is in what it lets you not think about. The fact that destructors may not be called is, unfortunately, something you have to think about.

In Rust, you cannot put something on the heap and "forget" to ever free it. Rust doesn't expose any mechanism in safe code to heap-allocate something that won't automatically free it for you when that thing goes out of scope (the primary way to do this is with the Box type).

As evidenced by this whole kerfuffle, you can in fact leak heap-allocated values by constructing an Rc cycle (or taking advantage of various implementation bugs in other things), or by using mem::forget() (which I believe is still marked as unsafe but seems likely to change). But none of these approaches could be characterized as "forget[ting] to ever free it", because they're not a consequence of "oops I forgot to do that".

I want to clarify a couple of points:

Most important, this API problem does not break Rust's basic safety guarantee. That is, you must write unsafe code to violate memory safety.

If Rust's basic memory safety guarantee were at risk, the core team would absolutely consider delaying the release, or doing whatever else was needed to address it! We must never allow memory safety violations in safe code, no matter how obscure the bug that leads to them.

So this is a question of writing "safe" APIs that hide uses of `unsafe`. What precise assumptions are you allowed to make within such code?

As it is, we are taking this API issue very seriously, and making sure that we explore all of the options available. We have already yanked the affected APIs, while we determine the best way forward.

There are known ways to safely re-introduce the (relatively few) places in the standard library where unsafe code was using this pattern. Gankro talks about one in the post; you can see a proposal here (https://github.com/rust-lang/rfcs/pull/1084) for the `scoped` API.

The basic issue here is about a tension between a couple of different APIs, as Gankro explained in the post. In particular:

- `Rc` has been part of Rust for a long, long time, and is a fundamental systems programming tool.

- APIs like `scoped` and `drain_range` would like to use RAII for ergonomics and consistency, but this is either not possible (`scoped`) or subtle (`drain_range`) given `Rc`. In either case, these APIs use `unsafe` internally, so it's all about what that unsafe code can assume.

Furthermore, there is not a strong consensus about how best to resolve these issues, even if we had all the time in the world. Currently there are at least three camps:

- Stick with the status quo (perhaps marking `mem::forget` safe to reflect that reality). Safe code can assume no leakage, but unsafe code needs to take extra care (and sometimes avoid the RAII pattern). This is already the case for a large number of other properties, as Gankro points out. Leakage in safe code can never lead to memory unsafety; it is just a vanilla bug, and Rust doesn't prevent you from making bugs.

- Introduce something like `Leak`, meaning that you have to explicitly ask for a given type to be guaranteed not to leak through things like `Rc` cycles. While that allows you to write APIs like `scoped` and `drain_range` easily using RAII, you need to be aware of the marker, and make sure to use it for such types. Worse, though, is the interaction with trait objects: depending on the design, you may have to write `Box<MyTrait + Leak>` to be able to store the trait object behind an `Rc`, and that `Leak` bound needs to be present for the entire chain of APIs leading up to that point.

- Restrict `Rc` in some way, perhaps to `'static` data, thereby ruling out (a class of) leaks for all types. It's not completely clear how much fallout this would involve. The compiler currently relies on non-`'static` reference cells, and this change would likely force channels to use a `'static` bound as well, thereby defeating much of the purpose of the `scoped` API in the first place.

Finally, it's worth noting that at least one version of the `Leak` proposal can be added backwards-compatibly, later on, so there is potentially plenty of time to explore that approach if we feel the complexity is worth it.

I believe that Niko Matsakis (also on the core team) is planning to write a blog post explaining all of the above in much greater detail.

Largely unrelated question: In the code about a third of the way into the article, there is,

    impl<'a> Drop for Foo<'a> {
        fn drop(&mut self) {
            *self.0 += 1;
        }
    }

What does the line "self.0 += 1;" do? Specifically, the ".0"? Foo is a basically a reference to an i32, so I would have thought that would be "self += 1;".

The .0 is a tuple access. Foo is a 'tuple struct' of length 1: http://doc.rust-lang.org/nightly/book/tuple-structs.html (though the syntax is explained in the section on tuples: http://doc.rust-lang.org/nightly/book/primitive-types.html#t... )

Ah! Cool. Thanks!

I really like the title and the first sentence. "Much existential anguish and ennui was recently triggered.."

This title made my day!

Reply on news.ycombinator.com