Using a Vec actually has some benefits beyond the safety guarantees from rust. There are fewer allocations, and you probably are more likely to get nearby nodes on the same cache line.
Yes, there is, just use plain old pointers. Who cares about UB anyway.