To complete the analogy, you'd have to want to ride your horse on a country road every so often in a way that breaks the law (original comment wanted to break the speed limit). Maybe you want to ride on private roads or something like that.
If your road and traffic infrastructure assumes no humans, you can do all sorts of neat stuff like have cars driving practically bumper-to-bumper for drafting, get rid of traffic lights / stop signs / etc. and have the cars communicate directly with each other to organise crossing busy intersections at close to full speed [1], and so on.
A single human driver in that kind of system would make every vehicle around them slow down and defer to the human driver's unpredictable behaviour and incredibly slow (compared to a computer) reflexes, with butterfly effects over a much wider geographic area... assuming the infrastructure for you to drive safely still exists.
Personally I don't think we'll ever get there... way too much infrastructure required, and a singly badly maintained car or bad actor could cause disproportionate amounts of havoc.
[1] like this https://youtu.be/ZhGduew_VjQ - the first ~30 seconds or so is the sort of thing you'd expect to see in a L5 FSD exclusive system... and then the last few seconds are what you'd expect to see when you suddenly add a human driver to the mix.
It's established fact that "just add more lanes" doesn't actually help.