If you have travelled to enough different countries, (60 for me and counting), you will have several instances of being detained by immigration, yelled at by consular officers, beaten up by border guards, have your passport flung at your face by police, and several such adventures. The fact that you went to great lengths to get the right visa papers (which could involve filling lengthy forms divulging every aspect of your finances and your life till date and wait months for a decision granting you permission to visit a country for a week) doesn't count. The fact that you have a PhD from a US university doesn't count. The fact that you have a great job at a fortune 5 firm in silicon valley doesn't count. The fact that you make a high income and have a lot of savings doesn't count.
In fact these could count against you, because immigration / consular officers will think someone with your passport is obviously making it up. That you are just trying to sneak into the country to get an under the table job washing dishes. This happens even when the country you are trying to enter gets a massive amount of financial aid from your own country.
So it's not a rich vs poor aspect. Traveling is a type of large-scale consumption that re-affirms self-worth through having the right passport.
You're thinking of "vacations", which are in fact expensive. Traveling in the fun parts of the developing world is really quite cheap.