See the shell script in the top comment here: http://libtorrent.rakshasa.no/ticket/2100

I tested it, and it works as advertised.

For firefox integration, in about:config create a new boolean as:

  network.protocol-handler.expose.magnet := false

For chromium integration, you need to use Xdg-open.