Yeah, that's on my roadmap for "llm " (hence the name) - I want to be able to use the same tool to execute against local models as well.

Everything that goes through the tool can be logged to SQLite so this should make it easier to build up comparisons of different models.