You don't strictly need known/consistent types, but it sure helps, since otherwise everything needs to be 8 bytes.
I don't think a way to read into and out of ArrayBuffers is possible, since these can have pointers in them. I think it needs a StructArray class instead, so there's a way to actually make a compact memory array out of all of this.