I agree that if that were the case, it would be a problem.
However, I don't think we will see a 6x slowdown here. JavaScript engines are already very fast on compiled code, because they have already been optimizing for typed arrays and so forth for a while now. Google even added a compiled C++ benchmark to Octane, for example.
In current benchmarks, asm.js gives you a speedup of anywhere from 1x (where normal JS optimizations already got us very close to native) to 6x. So 6x is the worst case, and there are plenty of cases in the middle. A 2x-3x difference is not big enough to make something a false sense of portability, the web has tolerated 2x-3x speed differences for a long time now, and these things also change a lot in terms of who is fastest, on what, and by what.
Furthermore, I would not expect a 2x-3x difference to last very long, again, because browsers have been competing on performance of this type of code for a while now (Octane, etc.), and will continue to do so.