Personally, as someone who has spent a lot of time trying to manage the safe/unsafe code boundary, I'm inclined to agree with you. Unsafe code in Swift has an even scarrier subtle hazard: if the last uses of a value are through an unmanaged handle, the compiler won't know and can free the managed value before those uses. You need to insert a marker saying "please keep this alive". Even worse: returning a value, after inlining, isn't sufficient to establish a use!
That said, your lock handle example doesn't matter in swift today: you really want a lock handle to be a struct, and structs can't have destructors. You also want it to have move semantics, which don't exist yet. This proposal suggests moveonly values that could adopt destructors (like Rust), and if you're already adding an annotation for destructors, it's not out of the question to add another one asking for "true" lexical destructors.