I love this idea! I agree that LLMs often don't have the knowledge that's actually important (a frustration I have with ChatGPT is that it remembers unimportant things and then over-indexes on them in its responses).
I'm happy to offer more feedback- my email is micah.blachman@gmail.com