I regularly (like daily) see empty cars (Waymo & Cruise) driving around residential areas by Lake Merced while I'm driving to pick up/drop off my kid from summer school... that's between the hours 8am and noon. Also see them in the Ingleside neighborhoods during the day: they have been completely empty when I've seen them. So maybe they can't carry passengers other than those other times, but they are definitely driving around outside of the hours you cite.

Having said that, I've not seen any bad driving from them. There are certainly far worse human drivers, motorcyclists, bicyclists, and pedestrians out there on a regular basis, so I'm not against the self-driving cars being on the street.