It's annoying because it's not like other permissions, where you can ask the OS to prompt the user, and check if the user granted it or not, but it's some special permission. If the user, by mistake because it doesn't know that it's needed, doesn't give it one time it's impossible to ask again, and the app doesn't have a way to know that the permission is not granted. It's just things that the customer service has to handle, and that is bad.
Sure, right to ask a permission, so make it like a regular permission as the location permission.