It's perfectly fine to like driving. However, any manual driving, and yours too, increases the chances of injury / death for other people. So, basically, pleasure drivers get their pleasure at the expense of other people's safety.

My solution would be to somehow segregate pleasure driving in such a way that people living their normal everyday life are not affected by it -- the same way pleasure shooting / fighting is segregated away from streets, restaurants and cafes to shooting ranges / fight clubs.