We used to use that, but unfortunately it's very buggy - we had many, many cases where a user would open a file with Kami, but the per-app permission would not be assigned by Google so it'd fail to open. For a while I had it so that we'd request the full drive access permission only if we hit that error, but in most cases it just meant we'd end up asking for permissions multiple times.
Probably the best option I can suggest if you don't want the permission is to use the open from local file button and choose a file from your synced drive folder or from the virtual drive folder on Chromebooks. You will lose the auto-saving to drive though.