We already have systems that permit pedestrians to cross with priority over cars; they don't need lights.

They don't scale to really busy streets, and one of the failure modes would be perpetually blocked vehicles.

And this still leaves other road users that aren't autonomous cars up in the air.