I agree, this was bothering me too. But they implemented it all or nothing. If I disable it in preferences it is completely disabled, no way to activate it for a single site.

They should have disabled the popup, but still allow me to activate it on a per site basis by clicking on the ⓘ in the url bar. Same for location, webcam and mic. I don't want sites to nag me with the location sharing notification, but on google maps I would like to enable it manually by pressing the ⓘ and granting permission.