You probably had to compile libsodium and build a shared object. That can take a long time. But the scripts that run when npm installing modules can contain malicious payload, yes.

I always try to find something that runs in web-assembly, but it's better to avoid nodejs altogether if you want high security. Go is much better for these kind of things overall.