The "harm" in this case is potentially a few seconds added to an elevator ride. I think there is another psychological effect to consider. Once you've boarded an elevator, you're no longer really 'waiting'. Even if an elevator takes you on a longer trip than necessary it might not be as annoying as a longer elevator wait. And ultimately, elevators are not ran by democracy. If the algorithm benefits thousands of employees in an office then any complainers will probably be told to shove it by the office staff.

Nobody likes being stopped at a red traffic light but we all understand they're a necessary evil so we don't crash and kill ourselves.