That's not overall in Brazil. I live in Sao Paulo and it's certainly not rude to arrive on time or closer to that time.
The time is usually a reference of "arrive after that time" and people will arrive in the hour or two after the time set. Usually closer friends will arrive earlier and not so close friends will arrive late because the party will be already full.