Best practice for doing this with the BEAM (the Erlang VM) is to just run two VMs, one for your app code and a “dirty” one that is allowed to crash and be restarted without bringing down your app VM.
If you're considering running a second (or multiple) BEAM for your nifs, you should also consider running your native code as a c-node or a port program.
A compelling use case is where your website and numerical engine are running on different machines. In that circumstance, your main application is already isolated from issues on the numerical engine.