If fontconfig fails to stat() a font file, it presumably aborts trying to record any information about that font file. Notably, it doesn't know what the font file's actual font name is. If all fonts fail due to stat(), no information will be available about any font. fontconfig has multiple levels of fallback (e.g. "similar" fonts first), but in this case since nothing is known about any font you just get whatever font happens to be first in the list of all fonts.
Honestly, everyone should run their filesystems with the `noatime` option set, so that they never record access times at all. We very rarely care about when files are accessed. Usually we care most about when a file was last _modified_, the mtime, so this loses us almost nothing and saves a huge amount of writes to the filesystem at the same time.
atimes aren’t completely useless, just useful less often than mtimes.