Apple already have this working on iOS - as discussed in this recent post.

https://unix.foo/posts/local-ai-needs-to-be-norm/