Making Clouds Go Away on MapBox Satellite | Hacker News Reader

Making Clouds Go Away on MapBox Satellite

mapbox.com

140 pointsincanus7727 comments

The maps look absolutely beautiful. Can I suggest you drop Apple a call and offer your assistance? I just came back from Singapore, and good portions of their hybrid maps were basically unusable because of cloud obstruction. (To be fair, Google also has a bit of cloud coverage of Singapore - but their hybrid maps were completely usable).

Cloud obstruction of satellite imagery six months after Apple released their mapping data, seems to indicate a fundamental lack of knowledge of how to deal with those type of obstructions, so hopefully your call will be well received.

Got a number?

Wouldn´t the sun-synchronous orbit allow you to use a similar a pipeline with local-time filtering to create a night-time map? That would be quite amazing, and also useful for socio-economic folks (or even correlate that with OSM coverage)

I could only find a related talk about this here: http://modis.gsfc.nasa.gov/sci_team/meetings/199905/presenta...

The images are amazing. When looking for clear skies, I was curious if you saved any metrics regarding how often a particular location (pixel) is clear or cloudy?

(NOTE: I have no intention of visiting the cloudiest place on earth with my vampire hunting kit. ;-)

You might find the weather simulation work done on the old (2005) "Earth Simulator" or newer (2011) "K Computer" supercomputers in Japan really interesting.

http://www.hpcwire.com/hpcwire/2011-06-20/japanese_supercomp...

I was curious if you saved any metrics regarding how often a particular location (pixel) is clear or cloudy?

We didn’t, but it would be pretty easy to re-derive. The cloudiest places at this scale are the sides of mountains in the Intertropical Convergence Zone[0], where wet air rises several thousand meters, thus cooling and thus condensing. For example, tomorrow I’ll be spending a lot of time looking at Andes in Colombia, Ecuador, and Peru.

0. http://www.youtube.com/watch?feature=player_embedded&v=Y...

This is pretty cool !

I worked on a similar problem during my bachelor thesis[1] : I wrote an algorithm to remove clouds from NDVI images (used for deforestation detection[2]). We used fourrier, moving windows and some ugly hacks to detect and interpolate cloudy data. There are some details in my report.

[1] http://ape.iict.ch/teaching/DiplomaReports/2009_Rebetez.pdf

[2] http://terra-i.org/terra-i.html

shamelessplug: Actually MODIS scans Earth using different wave lengths. The near infrared channel can be used to detect clouds and even to distinguish them from ice. Using a little recursive AviSynth Script I was able to produce an animation of daily changes in the European part of the Arctic with last year's data. It starts with all the clouds and then adds each day's cloud free pixels. The goal was to visualize how Kara Sea melted out within a few days. Enjoy: http://www.youtube.com/watch?v=j-EDFs8f_78 /shamelessplug

Cool. I hope they do it separately for winter and summer so as to give an idea of typical snow/ice cover.

Author here.

We’re doing almost the opposite – as little non-permanent snow and ice as we can manage. The reason is simply that we want to make this as useful as possible as a general-purpose base layer, and people usually want to see spring/summer growth. It looks subjectively right even in regions where it’s only around for a minority of the year.

If you’re interested in seasonal dynamics, I recommend Blue Marble[0]. It’s really good work – we’re in touch with some of the people who made it, and they’re sharp folks. But it’s half the resolution that we’re aiming for, and has some areas of distracting interpolation artifacts. Our main goal is accuracy, but we’re also going for aesthetics in a way that Blue Marble wasn’t.

0. http://earthobservatory.nasa.gov/Features/BlueMarble/

In the article you say you took the pixels without clouds. Just looking at how white it is, or something more complex?

This is too interesting a topic to get into right now – I’ll have to do another blog post or something. For now I’ll say: looking at whiteness alone gets you like 80% of the way there. It’s kind of amazing how effective it is by itself.

I suspect you could also factor in the variance between data samples. If a pixel is occasionally lighter in shade, you can assume that the lighter samples are due to cloud cover.

One possible naive approach could be to simply dispose of the lightest 90% of samples and average the remainder. You could reduce noise by blurring the pixels before applying statistical sampling.

looks amazing !

Will this be open-sourced one day ? Or maybe integrated into TileMill ?

This was what I was curious about -- let's say, as you work your way up a mountain, the bottom never has snow, the middle has snow 50% of the time, and the top has snow 100% of the time.

How do you decide where the border is? You're going to have a snow cap of some size -- what size is it, and how do you get to look both realistic, and consistent-looking, without artificial-looking borders?

I do like your idea of it being summer everywhere, though, hemisphere be damned! :)

In the current draft, exactly 81.7%. No kidding.

Thanks for response.

I think what you are doing makes sense for most purposes and I look forward to seeing the final layers.

I'm especially interested in snow cover because of a side project of mine (hillmap.com) that is targeted at backcountry travelers (ski tourers, hikers etc...especially winter travelers since it does things like avalanche terrain analysis in a canvas overlay).

Blue marble is cool and there are some overlays from NOAA etc but winter satellite photos good enough to answer questions like "is this typically an open snow field or heinous bushwhack" or "does this lake freeze over" would be really useful for planing trips in winter or colder climates.

If you guys (or someone) hosted the daily datasets it would be possible to do the processing client side in a canvas for a user defined time periods which would be an awesome tool.

Hillmap.com is a very cool project. Thanks for sharing.

Hmm. I don’t know of any specific product that does what you need (high resolution and near-realtime is usually an expensive combination), but I would definitely look through the National Snow and Ice Data Center’s projects: http://nsidc.org

This is interesting, but very low on detail. For example, was another modis product used for clouds or was it based on a simple EVI/water cloud mask? Is this MOD09A1, 500m or 250m? Lastly, why or how is this MODIS layer going to compliment existing Landsat or DigitalGlobe?

looks pretty cool. great work. I especially loved the fact that your mehod does not blur the edges significantly and the color profiles are pretty good.

I have a few questions you haven't mentioned in your blog and would appreciate it if you can answer them here.

is there some sort of public ftp server you can log in and get say the last 2 days' maps of any given coordinate ? and if so can you share the address ?

additionally, are the image coordinates that come with the images really that accurate that you can do pixel comparisons over time without doing any sort of registration ?

Wouldn't the clouds still interfere with the averages though? That is areas that have lots of clouds would seem whiter than they really are? It seems to look alright though.

They detect the images with clouds, then average the rest.

Ok that makes sense. I thought they were using this as a replacement for filtering out pictures with clouds.

Well, we’re actually filtering out pixels with clouds. That’s what makes it seamless. Or, really, it’s extremely seamy, but all the seams are very subtle and between individual pixels instead of larger regions.

How do you know the difference between a cloud and a snowy/white sand area of ground?

Rate of change, probably.

There is a lot of work done for this in the scientific community, there are many algorithms. (is http://modis-atmos.gsfc.nasa.gov/MOD06_L2/) The most basic way is to look at the reflectivity, which must be very high, and the temperature, which must be very low. In this case, however, they probably do simply throw out all the snowy/white areas.

Reply on news.ycombinator.com