Passenger safety concerns? If anyone would be able to stop an autonomous vehicle, it may be not always in the passenger's best interest. Could be OK if the passenger has an override, though - if they decide to tell the car to not comply with a legitimate stop request, it should be treated as if they'd be driving themselves and refused to stop.

And, obviously, emergency stop should initiate stopping, but the vehicle should do so only when it's safe. It's not an escalator that's safe to halt immediately and at any moment.

For an completely unmanned vehicle (driving to pick up or to recharge/park for the night) - probably no big deal.