That's the real issue, IMO. It doesn't matter if Chrome has Dart as long as 2/3rd the market is using different browsers.
The VM is neat too, but from the user's perspective, it's an implementation detail. It's a really nice development tool, because it lets you run and debug your Dart code straight from source instead of trying to deal with generated code.
But I think it's relatively less important that the VM be shipped to end users. I think Dart can be a great, successful product without that happening.
At least with dart2js I have just one "VM" to target - Google have sorted the x-browser issues for me. Imagine the peculiarities you'd find in Microsoft's implementation, for example.