You would get a lot of false positives, meaning apps that legitimately need the location. For example navigation apps or unfortunately apps on Android which use Bluetooth.
This really annoys me, because the dialog looks the same for apps that want to use GPS data and apps that use Bluetooth, and there is no way to grant just Bluetooth permissions.