A lot of Javascript's optimizations are fragile, as in they don't always hold true, so there's overhead determining if the assumptions still apply.
For instance, Javascript's hidden class optimizations are disabled as soon as someone "breaks" the assumption by doing something like accessing a variable using a dynamic array index rather than dot notation.
Javascript runtimes also try to infer types as you say based on how objects are used, but this is also a fragile optimization. For instance, if you use an assumed "int" in a string-like way the runtime re-assigns the tagged type. And it's hard to know whether you (or the library you're using) does so.
Those programs might not represent a real workload but TechEmpower's framework benchmarks are closer. https://www.techempower.com/benchmarks/#section=data-r16&hw=...
On most of those benchmarks the fastest JS framework is ~5X slower than the fastest Java framework.
And yeah the array optimizations are related to what we both mentioned above, but again the optimization is not durable. It only works sometimes, and incurs its own overhead.