If your vim is compiled with the correct flag for GUI operations (I forget which), you can yank/delete text into the * or + registers to make it available to the system clipboards. This won't bring the line numbers, and lets you avoid using the mouse.