Thanks for all the great suggestions! In terms of your existing 98% accuracy solution, can you point me to more details if possible?
I grabbed AlexNet as its a bit easier to work with and was readily available as a fine tunable model on the Caffe Model Zoo, but you're certainly right that VGG or Googlenet would give more accuracy.
For training on clouds vs snow and deserts, this is a bit more of a proof of concept for now based on the data we had access to (state of California). Someone could certainly scale this up using a larger data set and generating more annotation data with examples of snow and deserts to handle more edge conditions. They would probably want to build Mechanical Turk support into the annotation tool if they did.
Thanks for all the great comments! All of this is open source so contributions using any of these suggestions are certainly possible.
Best, Brad Neuberg