It's more about deepening the story I think and making it 'bigger' I guess or another type of story telling. You look at a fantasy map and there's typically places on the map not even mentioned in the actual story as you say, but that helps build the setting.
Look at the map for the lord of the rings for example. A good number of the places in that map are barely mentioned in the actual trilogy. Yet that map's become one of the most iconic in fantasy in general.
It's more about what the maps inspire in your imagination while you read. When some random fantasy city name gets mentioned, you can look at the map and get perspective about where it is in the world.
It's also helpful I think to keep the world consistent for authors. It gives them a frame of reference for scale and distance in their stories and likely helps keep track of all the random names and cultures and such that they make up for their stories.