Maybe you could separately map shared RefCells to guard the pages returned by the mmap wrapper? If the region is backed by a file, you'd need to check those RefCells on every IO access too, though.
Examples would be:
* read-only: &ROMemMap -> &[u8]
* read-write: &mut RWMemMap -> &mut [u8]
* write-only: fn set(self: &mut WOMemMap, i: usize, b: u8),
or more generally: &mut WOMemMap -> &mut [WriteOnly<u8>] where WriteOnly<T> has fn set(&mut self, T)
Now for the shared case, consider this: aliasing rules can be avoided with atomic operations, i.e. Arc<AtomicUsize> is shared and can be safely (atomically) read/written by multiple threads.
In the multi-process case, you could provide an atomic API, although we don't currently seem to expose byte-level atomics (likely not present on some platforms) so if you wanted to write a demo you'd need to use the unstable intrinsics atm.
FWIW restricted to single-threaded code, this results in the Cell get/set API which is not hardware-atomic but cannot overlap with other accesses to the same memory, as Cell doesn't implement Share so any threading abstraction will block you from doing any kind of sharing of Cells.
That is, you can share &[Cell<u8>] pointing to any bag of bytes that sits in read-write memory, and anything in the same thread can read or write to it, safely.
However, as you say, exposing a safe mmap version for libraries to call is a different problem with different difficulties. eddyb's sibling suggestion of teasing out the multiple entangled uses of mmap sounds like a good one, although someone would have to experiment with it to be sure.