I use Chaz to just talk to any LLM from Matrix, my normal chat client.

https://github.com/arcuru/chaz