I'm sure people have thought about it, it's just hard, annoying, and asking a lot of mostly unpaid OSS contributors. Many mod developers are high school / college aged.

Sandboxing Java code running in process requires ugly and obscure security APIs and restricts you to having to have a common modding API (Forge). Many mods use bytecode patching and would be broken completely.