When you deny them, roughly how often in your experience does this cause the app to crash or display an error?
EDIT:an "allow for 10 min" option would resolve this
XPrivacy does have the benefit of making it easy to provide realistic-looking fake data, which I believe the CyanogenMod team is against.
While I'm not exactly certain on how Privacy Guard works (I have yet to examine that code base), if a phone returns [] for a list of contacts and the application crashes...