It would most likely still be possible to measure that the cache lines were occupied, which is enough to leak (other) data by treating it as addresses.