launchctl unload -w /System/Library/LaunchAgents/com.apple.rcd.plist
will quit the process responsible for the madness (rcd, the "remote control daemon").You'll have to remember to re-load this thing if you want the default behavior. Or if you encounter other unexpected situations, to restore the default insanity.
It might be easier to run this app instead; then you have an icon in your GUI desktop environment and an app you can simply quit to restore defaults. Plus this app allows you to assign any app to the "Play" media event.