Ask HN: How do you give a local AI model long-term memory? | Hacker News Reader

Ask HN: How do you give a local AI model long-term memory?

2 pointsjavantanna4 comments

I’m running local LLMs using Ollama and hitting the usual wall:

small context windows + no persistent memory = hard to solve multi-step or long-horizon tasks. For those who have built serious local setups: How do you give your model persistent memory? Vector DBs? RAG? Fine-tuned adapters? Some kind of external state management loop? Or a custom “memory module” you wrote yourself? I’m looking for practical approaches that let a local model remember past steps, keep working on long tasks, and behave more like an agent with continuity.

View on HN

If you don’t want to reinvent all of this yourself, this is exactly the problem we’re solving at Ailog.

Most local LLM setups break down because people try to use the model as both the reasoning engine and the memory store. That doesn’t scale. What works in production is a layered approach: external long-term memory (vector DB + metadata), short-term working state, aggressive summarization, and strict retrieval and evaluation loops.

That’s what we built at https://www.ailog.fr . We provide a production-ready RAG stack with persistent memory, retrieval controls, grounding checks, and evaluation tooling so models can handle long-horizon, multi-step tasks without blowing up the context window. It works with local or hosted models and keeps memory editable, auditable, and observable over time.

You can still build this yourself with Ollama, Chroma/Qdrant, and a custom orchestrator, but if you want something already wired, tested, and scalable, that’s the niche we’re filling.

Happy to answer questions or share architecture details if useful.

but in my company we work with legal documents so our data is soo confidential and we can't use apis i need to set it op offline!!

I built an agent that has access to my diary, it has the ability to build hierarchical summaries of my diary, which help to compress context, I gave it tools to read pages, search using full text indexes and RAG (the former worked better, but I think it's largely because of limitation in my RAG implementation), it also has the ability to record memories (append to a specific markdown page). The latter are automatically included in the system prompt, when I invoke chat.

https://github.com/robertolupi/augmented-awareness/blob/main...

I use it mostly non-interactively, to summarize my past diary entries and to create a Message Of The Day (MOTD) shown when I launch a terminal.

thanks man i need to take a look to your code bcoz as you said hierarchical summaries i try to implement it didn't work for me like i am building a system which ocr pdf of legal contracts between parties so this way breaks when there is time to extract specific clauses as per contract

Reply on news.ycombinator.com