What he's talking about is putting yourself in a mindset of very low expectations to maximize your own happiness.

For example, if your baseline "expectation" of people is that they will be rude and shitty, then you won't become annoyed when someone is rude and shitty to you. And when someone is nice to you, they have exceeded your expectations and it makes you happy.

So what you perhaps "expected" is that people generally would not be rude, which led to your disappointment when you encountered a rude person, which ultimately just generated annoyance/unhappiness for yourself (i.e. their rudeness is static regardless of how it made you feel).

The point is not really to go around bleakly expecting everything to be shit all the time, but just in general, the lower your expectations are, the less power you give people to disappoint and upset you, and the more you appreciate people for exceeding your expectations. It's not really about whether it's "right" or "wrong" to be annoyed at them but just a mindset shift for your own contentment.