This is so cool thanks for sharing. I can imagine it’s not technically possible (yet?) but it would be cool if this could simply be run as a browser extension rather than running a docker container
The PoC is a bit outdated but it's here: https://github.com/brave/cookiemonster/tree/webext
https://github.com/mlc-ai/web-llm
https://huggingface.co/docs/transformers.js/en/index
You do have to worry about WebGPU compatibility in browsers though.
[1] https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/Web...