You should check out how Figma plugins work. They have blog posts on all the tradeoffs they considered.

What I believe they settled on was a JS interpreter compiled to WASM -- it can run arbitrary JS but with very well-defined and restricted interfaces to the outside world (the browser's JS runtime environment).