The problem is that the process is so opaque that it's hard to get rid of the nagging question of, "Am I being screwed somehow?"
You say you should know the value of the car and what you're willing to pay. How can you be sure that what you're willing to pay is reasonable? What if what you're willing to pay is a thousand dollars too high because you screwed something up? You can mitigate this with more research, but it's a lot of work compared to buying most things.
You're right that the buyer ultimately has the power (as long as the buyer doesn't need a car right now and has a reasonable choice of dealers in his area), but you have to wield it. This is not true in most areas. I don't need to do research and look up tips for getting the best deal out of my local Safeway. I just go buy stuff (after getting a loyalty card). My power works invisibly to ensure I get a reasonably good deal on almost everything. To the extent that there are tricks, it's simple stuff like "when there are two identical items from different brands, buy the cheaper one." When buying a car, you still have the power, but you have to go out of your way to wield it, or else you'll get screwed.
People want an experience where they just go in, pay, and walk out with a good deal. You can get a good deal from car dealers if you put in the work, but it's that very fact that you have to put in the work that people don't like.