favyen

Yes, I think this is part of it -- when the model sees a new low-res image, it compares it to patterns that it has already seen to estimate what that location might look like at high-res.

The other important part is that the model inputs many low-res images (up to 18, i.e. about three months of images) to produce each high-res image. If you were to down-sample an image by 2x via averaging, then offset the image by one pixel to the right and down-sample it, then repeat for two more offsets, then across the four down-sampled images, you should have enough information to reconstruct the original image. We want our ML model to attempt a similar reconstruction but with actual low-res images. The idea breaks down in practice since pixel values from a camera aren't a perfect average of the light reflected from that grid cell, and there are seasonal changes and clouds and other dynamic factors, but with many aligned low-res captures (with sub-pixel offsets) an ML model should still be able to somewhat accurately estimate what the scene looks like at 2x or 4x higher-res (the Satlas map shows a 4x attempt). The current model we've deployed does this far from perfectly, so there are some issues like figuring out where the model might be making a mistake and enabling the model to best make use of the many low-res input images, and we're actively exploring how to improve on these.

This shares ideas with burst super-resolution, see e.g. Deep Burst Super-Resolution [https://arxiv.org/pdf/2101.10997.pdf].

We plan to eventually add some real paid high-res imagery to the map just as a comparison, but for now you would need to look at the map at https://satlas.allen.ai/map (select Super Resolution) and compare it to a source of aerial imagery like Google Maps or Bing Maps at the same spot.

The marine infrastructure (offshore platform and offshore wind turbine) and super-resolution models only use RGB bands (B04, B03, B02), while the solar farm, onshore wind turbine, and tree cover models use 9 Sentinel-2 bands (add B05, B06, B07, B08, B11, and B12). With enough high-quality labels, the extra bands do provide slightly improved performance (1-2% gain in our accuracy metric, e.g. from 89% to 91%), but we don't have a detailed comparison or analysis at this time.

Also, all of the models input three or four images of the same location (captured within a few months), with max temporal pooling used at intermediate layers to enable model to synthesize information across the images. This helps a lot, definitely when one image has a section obscured by clouds (so model can use the other images instead), and maybe also when different images provide different information (e.g. shadows going in different directions due to slightly different times of day).

Thanks for pointing this out! We will look into fixing it.

I fully agree that errors in extracted data can lead to making incorrect decisions/policies. Even for applications where accuracy is paramount, though, I think error-prone models still have their uses:

- For applications that only need summary statistics over certain geographies, analyzing small samples of data can yield correction factors and error estimates.

- The data could also be combined with manual verification to improve existing higher-precision but lower-recall datasets (e.g. OpenStreetMap where features are more likely to be correct but also have less overall coverage).

The models we use to extract the geospatial data (like solar farm and offshore platform positions) from Sentinel-2 imagery are currently separate from the Sentinel-2 upscaling model, which is a more exploratory project.

We report the accuracy of the data at [1]; the Satlas project is quite new and we're aiming to improve accuracy as well as add more categories over time.

We expect the geospatial data will be useful for certain applications, but I agree that the upscaled super-resolution output has more limited uses, especially in its current state outside the US since it is trained using NAIP imagery that is only available in the continental US. We're exploring methods to quantify and improve the accuracy of the upscaled imagery.

Note that the model weights, training data, and generated geospatial data can all be downloaded at [2].

[1] https://github.com/allenai/satlas/blob/main/DataValidationRe...

[2] https://github.com/allenai/satlas