This sounds like a great example of where you might want a union (https://doc.rust-lang.org/1.56.0/reference/items/unions.html)
I still feel like this trick has it's place though, as an example take a look at where I stole this trick from, by matklad [0] and tell me what you think :)
[0] https://github.com/rust-analyzer/rowan/blob/d2c7843858da9d9e...