There are tools like llama-swap that make this pretty bullet proof and offer an OAI style API and compatibility for local Agents. I've been running flavors of OpenClaw and Hermes exclusively on local models hosted this way.
Adding llama-swap adds a fair amount of complexity that is unpleasant to debug and its documentation isn't very good (the only real doc is the sample config file).
Personally, I don't see the value in idle timeouts or time based eviction. It doesn't improve model load performance but does mean you're more likely to incur a load penalty for any request.
I generally use the same two models for my agentic workloads and they sit comfortably next to one another in the 32GB of VRAM on my GPU. If I need to load a larger model, llama-swap can easily eject both models if necessary.