It's called Software Fault Isolation. There is a link to the paper on it in Wikipedia.