One of the primary difficulties with writing LLM applications is that prompts are basically not composable, and any LLM library that modifies your prompt is going to be a nightmare to work with.
MEMORY EXAMPLE INSTRUCTION [COMPLETION]
it will basically not work to wrap it in a prompt that's structured
INSTRUCTION MEMORY EXAMPLE [COMPLETION]
I'm currently building something that leverages an ensemble of different LLMs depending on the difficulty of a task and ran into this issue.
Dolly V2 takes "###Instruction: <your stuff> ###Response" as the structure fed to the model where as GPT3.5 Turbo wasn't trained to treat that particular structure as important.
The nice thing is that GPT3.5 Turbo will just roll with the prompt structure Dolly uses but that only works in very large LLMs, I'd imagine I wouldn't get away with it in other 12BN parameter models.
But realistically this could look like taking the "INSTRUCTION MEMORY EXAMPLE [COMPLETION]" schema represented in a library and each adapter would transform it into
"MEMORY EXAMPLE INSTRUCTION [COMPLETION]" schema or whatever is needed by the different model.
> Instruction: Write a poem and them emit a structure that follows a schema named X.
> Completion: [map-schema X "roses are red, violets are blue"]
Conceptually this is basically just a function call where context is local to the function.