Not at all, you can encase everything on an “unsafe” block and live happy.

However you won’t have Rust’s guarantees of fearless concurrency and you expose yourself to other defects related to memory (free after use etc.)

If you understand the language, you even program much of the std libraries with unsafe code in them.

For more info you can consult the Rustonomicon.