As for performance, in lots of use cases it's not going to be a big deal. If you are super sensitive to performance issues then you can just wait, meanwhile everyone else gets to use the new feature. You have to start somewhere and waiting to satisfy everything usually ends up with doing nothing
Contrary to what a lot of people guess, boxing is actually a really good strategy most of the time. And, is indeed what many people are doing here anyways. The design supports a pattern that allows for non-boxing, and I expect that we will both supply an implementation for that with reasonable defaults, and that source generators will be a great way to augment this to get highly specialized impl strategies for non-boxing depending on the varying domain needs any specialized customer may have.
The C# compiler could do it to a degree, but there would be too many caveats to make it actually useful. Unless the JIT team has a change of heart, you're probably never going to see this.