Embedded stuff roughly falls into 2 types:
1.Superloop, for simpler stuff. No worries about atomic register writes, at most you have a few interrupts. You don't need mutexes here. The overhead will always be low but it gets very hairy quickly with the more things you try to do concurrently.
2. Rtos-based. Think microkernel but with just a thread scheduler. You might have a dedicated IO thread that handles messages from your worker thread, which may listen to a GUI thread. In this case, you will need a way to prevent multiple threads from trying to use the same resource, whether that is a pin, a configuration register, an interrupt flag that is in the same register as an exception flag and must be cleared atomically via a special shadow register, etc...
If you do all hardware touching through 1 thread you don't really need them, but otherwise, you absolutely need mutexes. And there can be a substantial performance cost to using them too much.
As an example, say you had a UART receive thread that fired from an ISR handler each time a byte was received. You could naively grab the UART mutex and read the new data, clear the flag, and deal with any error conditions, then release it again. Even a 100mhz MCU could have trouble keeping up with a trivial 100kbaud serial stream due to the overhead of using the mutex too much.
RTOSes do have other means for locking resources, but you would want to understand exactly what you're doing and minimize overhead as much as possible.
I am not at all experienced in Rust but it seems that touching the hardware directly could end up being a large pain point.