Google photos is simply refusing to let you use their app all unless you grant them access to view all of your photos.
I'm all for privacy -far beyond the average person- but this seems like one of those cases of uninstall it and move on if you don't agree with the product decision.
Fwiw: Not a fan of google. Currently on iCloud + Flickr for non-sensitive photos