Karma is not slow, but some of our tests may be. Wallaby just allows not to run them if we are not editing anything that affects them.
Cross browser issues are always a pain, I know what you mean. However, I have realised that I can write the bulk of my code without worrying too much about them.
When I'm writing the bulk of logic, especially with TDD, I found that it's more efficient to run tests as fast as possible in the fastest engine and care about running across all browsers much less frequently. Especially if you run tests as you type.
Later on, closer to the end of the iteration, when preparing to commit my changes, I surely run all of the tests in all browsers/smartphones/tablets that I support. Typically it just works, especially when your codebase is mature, you already know the majority of cross-browser issue sources and have something in place to avoid them. Even if it doesn't work, the issues are normally easy to solve. Overall, less context switching, more focussed work.
Anyway, it's just my experience (you can read more about the motivation behind wallaby.js in my blog post - http://dm.gl/2015/01/30/wallaby/) and we are planning to support more browser runners, or even just karma as a runner for wallaby, in future. Please feel free to vote (https://github.com/wallabyjs/public/issues/29) or create new feature requests.