I agree, and hence the unpopularity of being a private pilot. When you drive a car, the consequences and required training are minimal. If you don't know where you are, you follow the signs to the gas station and ask for directions. If you run out of gas, you put your hood up and wait for someone to stop and give you gas. The net intelligence required is about zero. Hence, the popularity.

When flying, not only are you responsible for getting yourself where you want to go, you are also responsible for not running out of fuel and crashing onto a family of four having a nice barbecue. This is beyond the risk that most people are willing to take. Learning about VORs and DMEs and NDBs and GPSes are not worth the ability to travel across the country 2x to 4x faster than driving, especially when the distances people want to cover are in th 2-6 hour range. 6 hours is not much time. Neither is 3h time. So why not drive the same vehicle you use every day to get to work? There is no risk involved and what are you going to do with the 3 hours flying you own plane got you? Drink beer? Meh. Much better than not accidentally murdering anyone.

(Not saying I'm happy with this, but it's the reality. If you want to fly an airplane, You Are Weird. I Am Weird, but I can't expect everyone else to be weird...)