I believe that permission is only needed for the Find My App itself (so that it can display the map correctly centered w.r.t where you are).

I don't think it affects finding that device from other devices.