In fact, it's against the App Store guidelines to refuse to work without a permission (unless it's central to the app).