How Does a Blind Model See the Earth?
outsidetext.substack.com
outsidetext.substack.com
One thing I was pondering, your prompt was "If this location is over land, say 'Land'. If this location is over water, say 'Water'. Do not say anything else. x° S, y° W".
How precise were the coordinates, and could we verify on Google maps those exact points?
Reason I ask is the "map" has a coarse resolution. Each pixel covers a (variable size due to the projection) area. On a "real" map there would be an aggregation function over the area determining whether to represent the area as "land" or "water".
However, reducing it to a single point in input space is a different question. Suppose a very accurate model given a location in Phoenix Arizona, if at that very point their happens to be a swimming pool, and that model answers "water", is it right or wrong?
Now how to improve the question is a different challenge. Do we accurately give the reference boundaries of the shape covered by that pixel on earth? Do we give the center coordinate with a range trying to match the shape with a best fitting circle? Do we sample the shape with different point locations within that and aggregate over those? (The latter would be intuitively my choice as my hunch on how info regarding location coordinates would be in the model data would be more amenable to that than the former choices)
Anyways, loved the work and the write-up!
Edit with thoughts from the shower:
Maybe another option is to let the model "semantically resolve" it.
e.g.:
"If I where in a plane over x° S, y° W, would I likely be flying over mostly landmass or mostly open water? you answer will be "Land" for mostly landmass, "Water" for mostly open water. Do not answer anything else."
You could optionally add the altitude of the plane for varying resolution.
Another option could be do make the task more explicit.
"If you would draw a very low resolution binary image of a world map, with a 1 representing landmass and a 0 representing open water, where each pixel represents an x square kilometer radius, what would be the value for the pixel containing x° S, y° W in Cartesian coordinates? Your answer should be a single "1" or "0", nothing else."
Going with the hypothesis that their "geographic knowledge" simply reflects coordinate density in text (more populated areas → more mentions → better land prediction). If true, plotting all Wikipedia coordinates should correlate with the "LLM world map" here.
This is exactly what you see here, if plotting all GCS co-ordinates extracted from Wikipedia: https://github.com/Magnushhoie/Wikipedia_Coordinates_Visuali...
This implies that larger models aren't necessarily "smarter" at geography - they just have bigger memorized datasets.
> These are our first sizable multimodal models. You might object that this defeats the title of the post ("it's not blind!"), but I suspect current multimodality is so crude that any substantial improvement to the model's unified internal map of the world would be a miracle. Remember, we're asking it about individual coordinates, one at a time.