I agree with you about that specific downside you mentioned.
Plus there's some other downsides such as likelihood that you might have to mix small amounts of Javascript as a second language into a WASM heavy solution based on another language.
However those are tradeoffs, and depending on the situation those tradeoffs can be very acceptable. A lot of enterprise software for instance isn't particularly concerned with those issues, and benefits strongly from the other very very significant benefits that can come from the other side of the tradeoff of avoiding the javascript ecosystem (not just language, but culture, tooling, churn and etc). Those benefits can include a range of things but I will just point out one theme which is increased stability and maintainability .. expressed in thing like type safety and refactoring tools, longer release cycles, less reliance on packages in favour of batteries included comprehensive language libraries, less pain with package upgrades, less pain with the canoninical way to use your framework de jour changing each year, less pain with prior art patterns being (gradually) rediscovered (and renamed) , and generally a culture that prioritises reliability and maintainability over the new hotness (or hotmess, depending on perspective!).
There's other situations, such as building the absolute fastest load time for a customer page, that tradeoff might not be appropriate. However to be fair there are other techniques for managing that, such as server side rendering and client hydrating of SPAs. Or gasp server rendered sites hehe with just a sprinkling of client interactivity if needed.
(Actually I think SPAs are usually not great at fast time to responsiveness, but that's really another discussion and lets not get sidetracked. The main point is the drawbacks are tradeoffs, and for many systems they are appropriate tradeoffs.)