And if cars are receiving 20 times more actual use, that would imply that there would be 20 times less cars sold
Actually, no it wouldn't. If cars are getting 20 times more use they will wear out much more quickly than they do now. That means cars will have to be replaced much more frequently. There would be fewer cars sold than there are now, but it wouldn't be 20x fewer.
The operating percent of a car will go from 4% to that 96%
This seems wildly optimistic to me. The driverless cars may be capable of driving around 96% of the time, but that doesn't mean they can be carrying people 96% of the time. No matter how efficient the system, if there are enough cars to handle peak traffic during the day, then a lot of those cars will be sitting around doing nothing at night.