Why can't we test observer memory directly using the double-slit experiment?

If we use a double-slit experiment with a which-path detector, and have an observer try to remember (or forget) which slit each photon passed through, why can't we use the resulting pattern—interference or not—as a readout of the observer’s memory state?

If no one remembers the which-path info, we expect interference. If someone does remember, the pattern should collapse to two bands. Why hasn't this been tested as a method to probe the interaction between human memory and quantum measurement? Feels related to Wigner’s friend, but focused on the memory-retention threshold.

Is this experimentally viable?