That's one of the main problems with self driving in general. It's a completely different system if 99% are self driving instead of 100%.
L5 is about transport, where everything can be 10-100x more efficient if everone is FSD. But if even 1% decide to take the wheel then it's instant gridlock everywhere.
If you want to drive, and have fun, we'll have racetracks for that.
You saying you want to take the wheel every now and then is like saying sometimes you want to ride your horse on the 8 lane highway. Uh... no. I'm sure it's great (well, in the analogy it is), but you ruin the whole purpose of the highway.
Or "sometimes I just want to drive on the traintracks". Ok... so all trains have to yield to you, inconveniencing maybe tens of thousands of people, after knock-on effects?