Did some testing and found that it boils down to three things: RNG algorithm used in the standard library, forced use of double precision floating point numbers (the case for OCaml and javascript), and like you mentioned, memory management.

EDIT: forgot to mention the obvious things: compiler/interpreter maturity and inherent overhead.