You're holding it wrong if you're dragging files around a desktop to open them.

Just use a fuzzy file finder within the terminal or within vim itself.