The answer is in the question - the "polite response" is a polite response.
Bit if you're just looking for validation from the internet to not be polite, then here you have it - it's ok if you don't respond to a recruiter who you don't want to respond to.