What are your use-cases for image search? Mine are pretty straightforward; find a higher quality/uncropped version or to find the source (article, recipe, or photo set). Use case #1 I click through save and exit. Use case #2 I engage like I would for web (text) search.

#1 might be confused for not finding the intended result, but could be discerned after looking a bit more.

Also, if a lot of people add in "-site:pinterest.*" (or click on pinterest sites but come back, then find the intended result) that might be a hint it's not adding real value.