I think it's either running the model in the browser or a small part of it there. Maybe it's downloading parts of the model on the fly. But I kinda doubt it's all running on the server except for some simple RPC calls to the browser's WebGL.
Use client resources instead of server resources.
1. Load the page
2. Disconnect from the internet
3. Try to use the app without reconnecting
Edit: Good call! It's fully offline - I disabled the network in Chrome and it worked. Says it's 176MB. I think it must be downloading part of the model, all at once, but that's just a guess.
The 176MB is in storage which makes me think that my browser will hold onto it for a while. That's quite a lot. My browser really should provide a disk clearing tool that's more like OmniDiskSweeper than Clear History. If for instance it showed just the ones over 20MB, and my profile was using 1GB, at most it would be 50, a manageable amount to go through and clear the ones I don't need.
Google has started addressing this. I hope it becomes part of web standards soon.
https://developer.chrome.com/docs/ai/built-in
"Since these models aren't shared across websites, each site has to download them on page load. This is an impractical solution for developers and users"
The browser bundles might become quite large, but at least websites won't be.