I wonder how Porsche and the alike will adapt to that situation. Sure it will take some years before someone will start pushing it...
When level 5 self-driving is available, it won't be too difficult to add a wheel back and have modes where the self-driving capability maintains safety for other drivers on the road, but allows the driver to assume some risk for their own car and personal safety.
Different modes could allow the driver to accept different levels of risk to themselves and of their car, while still providing automatic protection and avoidance of other drivers on the road. Probably some more... extreme... modes could require posting a bond with the insurance company or government to cover personal injury.
If someone doesn't mind risking their car (collision insurance would no longer apply in these modes), and can pay for any injuries they sustain, as long as they aren't risking other people's health and safety, the computer could let them drive how they want, within certain risk limits.
I'm not so sure people will bother, though. Once car-to-car transponders are mandatory, creating an intelligent grid of vehicle awareness, what fun is it to zoom around other cars as if they're stationary if they know you're coming?
I don't believe people want the actual risk associated with aggressive driving. They just want to beat other drivers. Computer games should take up most of the slack. Even really simple games like Race The Sun offer plenty of adrenaline and reflex development with no actual risk. If you want actual (substantial) risk, go rock climb above your ability, or skydive without a reserve chute.
Having said all that, even if we're talking only about "manual modes" in l5 rather then "manual cars", there are still reasons why they might be banned or at least massively restricted in general, and that's due to infrastructure economics, not crashes. Well developed L5 cars will be capable of operating with a level of precision (in terms of actual road lines and such) and reliability that is beyond the general population, so once all cars reach that level it significantly changes how roads and support infrastructure themselves can be designed. Purely self-driving vehicles simply won't need the lanes to be as wide for example, which directly translates into newly available free space on exisitng roads. That could be used to make them purely thinner (less paving = massive cost savings), gain an extra mechanized vehicle lane for free (higher capacity for the same price), maintain existing numbers of lanes but gain new lanes for walkers/bikers, or some combination. There will be a lot of minor to radical optimizations available with significant economic and quality of life benefits, so long as you don't have to worry about having human drivers at all. That's probably going to tempt a lot of jurisdictions sooner or later.
Wide-scale fully automated cars will change everything. First, I think fewer people will personally own cars, perhaps a small fraction. Vehicles will need to be less “general purpose,” if they’re not personally owned. One or two seaters, low speed electrics are better/cheaper if you’re getting around a congested city on your own. If we don’t own our own, we don’t always need cars that can fit a family, handle highways…