Only if you assume everyone commutes to work at exactly the same time. Rush hour _is_ a thing, but even during rush hour you're no where near the point of having 100% of cars on the road at the same time.
You're also forgetting that most existing cars on the road have way more seating capacity than needed, since they need to account for the maximum number of people who will use that car at once. A family of four needs a car with four seats, even if most of the time that car will only be carrying one person to work and back throughout the week. Self driving cars will be much more efficient in that respect.