It would be better to store all the Client Secrets in such a Secrets Manager right? Or should I create an encryption key per user and store that in the Secrets Manager?
Whether you use one envelope key or one data key per client, one data key for all, or one envelope key for all is really a judgement call and depends on how paranoid you want to be vs. how much you want to worry about juggling keys.
Personally I don’t go per-user, but if you have the concept of a company/tenant/etc. I might do one for each of those. If you’re storing each tenant in a different database or region it lends itself more towards having a separate key for each as well.
Specifically I would think the decrypted data key is derived from the encryption key.