Downvotes could apply just to future recommendations for search results you see, and not apply to advertisements.