I had a similar idea some time ago but didn't implement due to its complexity and the need to juggle different parameters. On top of that, I don't think there are any guarantees when it comes to privacy. Your users will have to trust that no mistake will be made in handling the raw data or the preprocessed data and that no malicious actor will be able to access the original weights.

You should instead try looking into Homomorphic Encryption:

https://huggingface.co/blog/encrypted-llm

It is resource intensive and slower but it serves your purpose better, in my opinion.