I ended up turning memory off due to it frequently recording mundane aspects about the current chat session that would have either no value or negative value in future interactions.

"User wants to sort the data x and then do y"

Like, don't save that to your long term memory! I attempted to get it to ask me for permission before using its memory, but it still updated without asking.