Apps with a config file will often try to read `~/.config/myapp` and also `~/.myapp/config` and maybe one or two other places.
How many permission prompts will users tolerate?
If I drag&drop a file in an app or it's icon, or by right click. The permission would be implicit.
For configuration files they could be part of allow rules, since they are so common if they follow the XDG specification.
For saving files, flatpak apps could be allowed to save anywhere in the users home sub-directories (not home top-level) as long as it doesn't overwrite any existing file. There are downloads, public, documents pictures, videos, that are barely used by default.
There is plenty of room for better desktop based integration, while of course lacking funding and care for users (thinking of particular devs in this case).
Currently it's a mishmash of separately using flatseal (and equivalents) and having your fingers crossed if you're likely to exfiltrate or not your data.
If the file picker enumerates N different directories immediately (which aren't the active one) that causes a problem yes. I guess allowing enumeration access _anywhere_ (but not file read access) isn't a necessarily a problem.