People used to not using the buttons during the day may not remember to use them at night. They may not even be able to see the labels at night, since (unlike any actual road sign, at least in the US) they aren't even retroreflective. This is likely to cause confusion, leading people to cross without a walk signal.
So this will disproportionately endanger people walking at night--which is already the peak time for driver-inflicted fatalities.