Your product might still work if it has a second of lag after every operation, but I won't want to use it. And if there aren't any decent alternatives, I am going to experience irritation and sadness on a continuous basis.
Jonathan Blow said[0] (5 minute interview) that software authors have an ethical responsibility towards their users. How many people use YouTube, or Windows? The time wasted might be small on an individual scale, say a minute a day, but multiply it by a billion people (eg. YouTube or Windows) and that's 16 million man-hours per year.
I'd argue the same responsibility exists just as strongly even if you have one user! Do you want to frustrate her, constantly waste her time? Of course not! She is not going to be very happy with you if you do that :)
[0] Jonathan Blow on Success and Ethics in Software Development https://youtube.com/watch?v=k8gIJOy0c2g