For instance, you can see this in north-american films, where the mexicans are hired to play Brazilians, where we (brazilians) can spot right-away they dont look like any type of average looking brazilian. But im pretty sure it make sense for the crew of the film and people watching it, even when they speak spanish.
Just to give you a glimpse of famous brazilians north-americans may know: Rodrigo Santoro, Morena Bacarin, Alice Braga, Gisele Bundchen, Alessandra Ambrosio, Jordana Brewster, Camilla Belle, Kaya Scodelario, Wagner Moura.
The thing is, i've just give you this sample, so you can see how they don't have any resemblance between each other or homogeneous look. The ones that speak perfect english can perfectly act in roles not meant for "latinos" and nobody notice it.
I think nowadays is right to use this term in the same sense as you use for Europe to define France or Italy as part of Europe. Albeit is even wrong to use in the sense of a continent giving America is just one continent, with north and south.
Anyway if you dig down and look to all the inconsistencies of the term it boils down to some sort of racial profiling thing, because it makes no sense in any other way you look at it except for labeling it as "not white".