on the main topic, have you heard about RWKV ( rnn based gpt style network )? the project's actively working on implementing "infinite" context length support, which would probably pair very well with a project like yours
Reply on news.ycombinator.com