As self-driving highways, with cars that move at 250 km/h (155 mph), become common, regular driving will probably be restricted to secondary roads first, tertiary later. As time goes by, you will be seen as a nuisance from most of the drivers, as your car will be restricted to driving 60 km/h (37 mph) less than the rest in the same kind of road.

So by the time a law passes that forbids non-autonomous driving, it won't be a "fascist government" but your own countrymen who will take your "basic freedom" away.

But don't worry, I was using the "you" figuratively, by the time this happens, I don't think any of us will be alive. And, of course, there is always the off-chance that someone codifies in the constitution the right to drive a car, assuring that future generations have the freedom to kill themselves in the roads as we proudly do today.