Not Kyle, but another of the team involved in the project. When we built the training model, we used OpenStreetmap data to find locations of ~1,000,000 "things". A thousand churches, a thousand water towers, a thousand playgrounds, and so on. For each of those locations, we downloaded a satellite image.
The neural net was then trained to look for the things that make each one of those things distinct; what makes a playground different than a church? It could be patterns, it could be colors, it could be any number of things. (For more precise details, you'll need to talk to Aman or Kyle.) It compares lots of things to lots of things, makes some guesses, and then sees whether those guesses help it correctly determine what we told it was in each tile.
Once the model is trained, it's identified the 1024 "features" that are most significant in correctly distinguishing types of things from each other. We then run every tile of a geographical region through that feature determiner, which converts each tile into a point in our 1024 dimensional space. The search function then identifies a tile, looks up its location, and finds the 100 things closest to it within the 1024 dimensional space.
So, TL;dr: It's not looking for colors, it's looking for computable features, which may or may not be color-specific. (Actually, they're highly non-color-specific: the training model randomly "wiggles" the color to makes sure that it doesn't get too tied to a very precise color.)