You'd also have to fit the past interactions in the context window of the LLM, otherwise it wouldn't remember them.

Fine-tuning individual agents in order to move memories from the context window to the neural network weights, even if possible, would probably get too expensive.