libvips also have a block_untrusted mode where it will block unsafe loaders, .mat seems to be marked as untrusted:
vips -l
VipsForeignLoadMat (matload), load mat from file (.mat), priority=0, untrusted, is_a, get_flags, get_flags_filename, header, load