I think you misunderstand “Right of Way”. It’s not just an option for a pedestrian to cross, it’s incumbent on them to maintain predictable movement through the intersection.
Giving up Right of Way creates more confusion (think about the “go” / “no you go” game of a car not moving when it’s their turn at a 4-way stop).