That's one of the main problems with self driving in general. It's a completely different system if 99% are self driving instead of 100%.
L5 is about transport, where everything can be 10-100x more efficient if everone is FSD. But if even 1% decide to take the wheel then it's instant gridlock everywhere.
If you want to drive, and have fun, we'll have racetracks for that.
You saying you want to take the wheel every now and then is like saying sometimes you want to ride your horse on the 8 lane highway. Uh... no. I'm sure it's great (well, in the analogy it is), but you ruin the whole purpose of the highway.
Or "sometimes I just want to drive on the traintracks". Ok... so all trains have to yield to you, inconveniencing maybe tens of thousands of people, after knock-on effects?
If that were true we'd never get to L5. We'd need a hard switch over before L5 was deployable, and L5 needs to be deployable before there would ever be a hard switch over. Deadlock.
The hard truth is, self-driving is going to have to share the road for decades before human drivers are phased out completely, and, that being the case, there is no reason for them not to have steering wheels for the time being.
But also you can just say some streets are for FSD only. Some neighbourhoods.
Similar things are already rolling out as a soft barrier for the worst polluting cars. E.g. London's ULEZ. And other places ban studded tires, sometimes by individual streets.
You could argue "but I want to be able to actually stop on ice patches, so I want studded tires". And the answer is "fine, but then you can't drive your car on these specific areas". And likely the areas will expand.
It's like "sometimes I want to roar with my obnoxious harley". Hopefully in the future the answer is "that's fine, but if you do that in a city we'll throw you in jail… and then fire you from a canon into the sun", because it's not your freedom to wake up 50'000 people just to have some fun.
Or maybe it's like saying you want to ride your horse on a country road every so often.
If your road and traffic infrastructure assumes no humans, you can do all sorts of neat stuff like have cars driving practically bumper-to-bumper for drafting, get rid of traffic lights / stop signs / etc. and have the cars communicate directly with each other to organise crossing busy intersections at close to full speed [1], and so on.
A single human driver in that kind of system would make every vehicle around them slow down and defer to the human driver's unpredictable behaviour and incredibly slow (compared to a computer) reflexes, with butterfly effects over a much wider geographic area... assuming the infrastructure for you to drive safely still exists.
Personally I don't think we'll ever get there... way too much infrastructure required, and a singly badly maintained car or bad actor could cause disproportionate amounts of havoc.
[1] like this https://youtu.be/ZhGduew_VjQ - the first ~30 seconds or so is the sort of thing you'd expect to see in a L5 FSD exclusive system... and then the last few seconds are what you'd expect to see when you suddenly add a human driver to the mix.
It's established fact that "just add more lanes" doesn't actually help.
I wonder if someone made a research about likeness of reckless driving vs closeness to race track.
Like, if it was $5 to have a lap nearby (as in "not have to reserve day to go to track and have a track day") I wonder how less of dumbass driving we'd had on the streets...