Dirty schedulers isolate the potential scheduling issues, but there's no memory isolation between NIFs and the rest of the BEAM state, so there's danger there.
Best practice for doing this with the BEAM (the Erlang VM) is to just run two VMs, one for your app code and a “dirty” one that is allowed to crash and be restarted without bringing down your app VM.
If you're considering running a second (or multiple) BEAM for your nifs, you should also consider running your native code as a c-node or a port program.
A compelling use case is where your website and numerical engine are running on different machines. In that circumstance, your main application is already isolated from issues on the numerical engine.