You might be able to switch to Bun or Deno with relatively little effort and create binaries that are probably a lot smaller (and maybe also faster) with each of their compile commands [1][2].
In the end I do also believe that a compiled language like Swift, Rust or Go might be a better fit though.
[1] https://docs.deno.com/runtime/reference/cli/compiler/ [2] https://bun.sh/docs/bundler/executables