Particularly egregious was (is?) async/await. Upgrade your browser/runtime/don't use it you say? Sure, but first two weren't always possible, and the third isn't possible unless you thoroughly vet your dependencies (easier said than done).
"Compiling to javascript" is all well and good if you actually just compile to normal javascript, as soon as you have any code that simulates other features (classes/objects/what-have-you) you are no longer "compiling to javascript". I mean yeah sure as a sort of intermediary assembly language you are but the performance is not the same. You have a new language with a runtime overhead, that now requires you modify the "core" language to bring in new features, which results in the underlying execution engines (browsers/cpus) becoming more complicated, power hungry, etc....
Anyways, type caching is not all bad. While the TS overhead is likely responsible for the performance wins for javascript in the following chart: https://programming-language-benchmarks.vercel.app/typescrip...
The performance wins for typescript likely source from the ability of the runtime to pre-allocate and avoid type checking.
Providing the type checks without using any non-JS features (and possibly providing the runtime some heads up regarding checks to safely drop) is the ideal.