something like
---
"You are Bob the Farmer. You grow rutabegas in the Kingdom of Foo. You are a cautious, fearful man. Several years ago your home was pillaged by King Foo and your family was taken. [blah blah blah several more paragraphs of biographical information]
Your primary motivation is to get your family back and keep the farm going so that you don't starve.
Last week you saw a mysterious group of figures in the woods who appeared to be headless. This is bothering you, along with the stress of your missing family. You wish a band of strong warriors could investigate, especially if they have a necromancer among them.
You may discuss any of the general world knowledge in background_lore.txt
You know nothing about the following topics: [[insert list of plot points that haven't happened yet or are unknown to him]] and will become confused, fearful, and slightly belligerent when asked about them."
---
You could of course update the system prompts for each character as the events of the game progress.
It would be a lot of work to keep these system prompts updated and relevant, for every character, as game events progress, but I think some of this could be handled by some kind of inheritance system.
Suppose "Bob" lives in "Town A", which is in "Kingdom B." You could probably define a lot of things at the Town/Kingdom level. Like suppose "Kingdom B" is plagued by orcs, but "Town A" is kind of a citadel that is safe against orcs. "Bob"'s system prompt could inherit a lot of concerns and knowledge from "Town A" and "Kingdom B"... the system would not have to be strictly hierarchical either.
Not every NPC would have something deeper to offer, much like not everyone in our world would have something that would pique my interest (in a general sense -- I'm sure I could learn something from anyone I spoke with), but it would also make me interested in conversations with NPCs at a deeper level than I currently engage with.
But of course there is a story behind them.