I would respond to your request with "no" instead of ignoring it.

Saying no isn't rude. Ignoring that someone has spoken at all is, in most cases.