I expect that as the automated cars get sophisticated enough that they no longer act individually, but rather each car is in constant communication with the cars in the area so that they can coordinate maneuvers and engage in global planning, a semi-manual option might be added to the system so people can drive for fun.
They way that would work is that when you car is in semi-manual mode, it tells the others cars that a human is controlling steering and speed. The other cars will then give the human extra room (e.g., no drafting or close formation driving on the freeway, no crossing at right angles in an intersection at full speed just inches apart, or that kind of stuff that the cars will do when computers are driving all of the cars in an area).
The automated system will still be monitoring the human and can take over if the human does something bad, so the other drivers don't have to worry that a human driven car will suddenly cross the line into their lane and kill them.
This could even be turned into a game, where the automated system can score the human driver based on how well the human drove and how often the system had to intervene to keep the human from crashing.
Note that racing could also be allowed in such a system. Want to take that Aston Martin out for a race against your neighbor's Ferrari? Let the system know, and it can give you a course through the city, clear a bubble around you two, and let you go at it.