They're allowed to have limits like "local driverless taxis don't operate outside SF city limits or below 35 degrees with precip in the forecast" etc. at level 4, but to meet level 5 (per the bet) it has to be able to "drive everywhere and in all conditions," [0] which adds a lot of really difficult edge cases.
Situations that come immediately to mind:
- Driving in the hurricane lane on the shoulder during an evacuation
- Reversible lanes and streets
- Sizing up an icy hill and figuring out whether it's safe to keep going
- Ferries
- Knowing a baseball entering the road from behind a parked car will probably be followed by a child
- Understanding traffic police, sign turners, "follow me" trucks, etc.
[0] https://www.sae.org/binaries/content/assets/cm/content/blog/...