popular LLMs for public use already do this. chatGPT with a pro account remembers context from previous interactions.

It's simple really they just have the context inserted as part of the input. It's part of the prompt. The prompt is invisible to you as a user.

input:

   "You are an LLM with long term memory saved as json objects. When talking to the user you will receive a query from the user and some json memory as context. Add to this memory as you communicate. If the json object exceeds 4000 entries trim 100 entries that seem less important. The users query begins now:

   {}

   Hi my name is Bernard. 
   "

Output:

   "
   {"user name": "Bernard"}

   Hi Bernard, How are you?
   "

Context is extracted from the output and is given as input into other conversations.