Your best and safest bet is not to need this feature in the first place.
Failing that, what you're probably looking for is something along the lines of a simulated HSM (let's call this an SHSM).
Decompose the system into four distinct services:
- The front-end application, "Mint" as experienced by normal users.
- A minimalized subset of "Mint", with Mint's chrome and UX, dedicated to the acceptance of credentials or secrets from users. Ideally, this is running on a separate instance.
- An SHSM system that exposes nothing but a simple remote API for sealing and unsealing secrets. Seal: secret->token; unseal: token->secret. The SHSM is on separate hardware, unshared by any other service.
- A backend system, hardened and carefully assessed (but not as hardened as the SHSM) that performs the operations that require the secrets and stores its results where the front-end can get access to them.
This design needs to recognize that:
* The minimalized secret-accepting front-end can be compromised and, if that happens, an attacker can "camp" n the front-end and collect secrets regardless of what else you do. That's why you segment this functionality from the main front-end.
* A serious flaw in the rest of the front-end might transitively give an attacker a pivot to owning up the secret-accepting front-end; this could be direct (for instance, if both share a database) or indirect (if the minimalized front-end renders attacker-controlled content accepted in the front-end).
* There needs to be a diode-like relationship between the secret-acceptor and the SHSM; the SHSM must allow the secret-acceptor to seal a secret in such a way that it can't later unseal it, else it's serving no real purpose (an attacker that compromised the secret-acceptor would be able to unseal all secrets).
* The SHSM itself must be locked down to exactly the minimum set of services required to perform the seal/unseal operation.
* Ideally, the security of the SHSM asymptotically approaches that of a real HSM; for instance, a hardware-augmented root-of-trust might be used to make it difficult for an attacker that (improbably) gains code-execution capability to instantiate their own SHSM process to decrypt secrets.
* The backend service, which might have unlimited access to the secrets stored in the SHSM, is itself a prime target of attack through clientside vulnerabilities (this is why you want to segment it off from the SHSM). Some combination of rate-limiting, anomaly detection, privilege, and user-permissions (a system might be designed to prevent the unsealing of a secret without some action having happened in the front-end within a window of time).
None of these problems are easy. You might plan on spending somewhere between 4x-10x the amount of time designing, building, verifying, and monitoring this system than you would a system doing something similarly complex with non-secret data.