no? A file system is the format that the data on the disk is stored as. If you mount an ext4 disk as ntfs, it wouldn't load properly. It's not just the interface for loading the data, it's how the data is actually stored.
I think that it's a pretty good heuristic that if permissions exist in the filesystem, they matter and shouldn't be ignored.
But I agree with you, they do have a use and to some use cases matter, and we shouldn't arbitrarily decide to ignore them.