The first is that value types themselves are immutable. This affects code generation and optimization. If you were to modify the value with unmanaged code then you may not observe the modification properly from managed code. Maybe this restriction will get relaxed, but I don't see that on any roadmap any time soon.
The second problem is that value types are still nullable. The flattened array is not going to be identical to a Go slice or a C# Span etc. because it has to track the nullness of each element. It seems they don't want to nail down the exact storage format for that yet, possibly to change it in the future, and possibly because they want to add language-level control over nullability eventually too.
If you cannot get a reference, it's possible to maintain mutability, but all access would involve copies. But it's not clear that this is what most people want because it's rather similar to what you can do with library-based solutions today (involving ByteBuffers and whatnot).