But brute force everything you get back? Easy.
Take every key-sized chunk of heap you got back, and see if, when interpreted as the key, it is the private key to the public cert. When found, send it in!
edit: OpenSSL wraps malloc, so it is different on different systems. You could find out what the probabilities are by looking at the source. For some reason FreeBSD's malloc gave up private keys with the default exploit length and not much effort. It was only a matter of time before other platforms were also figured out.
Note that the FreeBSD exploit worked on a fresh server after boot, which might have also been the case here.
Note that in the heap there are the intermediate values that and are used when constructing the connection and doing the encryption, you aren't exactly looking for '------BEGIN PRIVATE KEY------' or base64 string, you'd be looking for those - each which have a unique data type.
That's what I thought as well when I saw people POSTing strings like that and other people finding them. But when the key is first loaded from file, wouldn't actually all those base64 strings be in memory?
Probably just pretty unlikely that they would persist for a long time without being overwritten.
You can't, however, get away with not having the private key modulus in memory (in some form) all the time.
[0] http://blog.existentialize.com/diagnosis-of-the-openssl-hear...