Hacker News Reader

Yup, we added this feature to Varnish Cache a few years ago, random key encryption. It generates a random key at startup and encrypts all memory with it. Since this kind of memory is only resident for the lifetime of the process, it works. We stored the random key in the Linux kernel using the crypto API [0] just because its not safe storing any kind of keys in a memory space used for caching (Cloudbleed [1]). We then use the key to generate a per object HMAC, so each piece of data ends up with its own key, which further prevents something like Cloudbleed. Since we used kernel crypto, overhead was about 50%. If you stay completely in user space, its probably much lower.

[0] https://www.kernel.org/doc/html/v4.17/crypto/userspace-if.ht...

[1] https://en.wikipedia.org/wiki/Cloudbleed

Could you point me to the relevant source code? Am highly interested to take a look at it during the weekend.

Closed source, write up would be here:

https://info.varnish-software.com/blog/introducing-varnish-t...

> Closed source

Ah, so we'll just have to trust you that it's doing anything at all, then.

"Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith."

https://news.ycombinator.com/newsguidelines.html

> "Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith."

> https://news.ycombinator.com/newsguidelines.html

Forgive me but can we not be skeptical of claims made about a commercial product?

Of course you can, and there are plenty of ways to do so that don't break the site guidelines. Cheap, snarky one-liners are not the way. If someone's posting about their own work, there's no need to be disrespectful.

It's also not helpful to post such a clichéd dismissal of what someone else says or their work. That's in the site guidelines too.

https://news.ycombinator.com/newsguidelines.html

Isn’t his reply valid and an exception to the rule given the context?

We have no problem sharing our codebase with customers, especially if there are concerns like this. Shoot me a msg if you are genuinely interested in anything you have read.

No offense, I am genuinely curious, why would anyone use any closed source software for anything related to security after the Snowden revelations?

Just to clarify my understanding, the reason for doing this is so that random sampling / leakage of the contents of the RAM stops being useful, you need to specifically get the key (and then presumably a whole chunk of encrypted RAM to decrypt?)?

Yup. When something goes wrong in these kinds of applications, you sometimes tend to just randomly dump memory, which is a huge data leak. Or even worse, if someone figures out a way to force a data leak, then your are completely compromised. Having each piece of data with its own key and that key is a combination of data outside of the process address space drastically lowers the chances of data leakage and total compromise.

Could this be implemented at the OS level, i.e. whenever a proces launches, the OS generates a key that it will keep to itself and use to transparently encrypt all memory allocated by that process?

Technically yes, but practically no, because mediating all memory reads through the kernel would be very slow.

SME/MKTME add hardware support for this.

You'd probably want a hardware module to do that lest performance plummets. Memory controllers can already deal with ECC efficiently, adding a simple cypher on top of it should definitely be feasible.

Possibly, but memory is accessed using plain CPU instructions, so it would be hard to transparently encrypt all memory for an application at the kernel level. You do have virtual memory, but I dont think that could be leveraged for this. But who knows whats possible there, maybe if you align and address each memory value at the page boundaries and always force a page fault you could have a really poor implementation :)

Transparent disk encryption, not a problem since devices have filesystems which can implement encryption at that layer.

But that wouldn't prevent (mitigate) cloudbleed anymore as the problem is about isolating contexts within process boundaries.

Modern Intel chips can encrypt memory on the fly without performance loss (SGX does this). However I think it's not exposed for non-enclave use. Perhaps it should be.

Note: inside the enclave there is a performance loss but that's due to MAC checks. If you just want encryption without integrity against tampering you don't need that.

Yes. Most research makes CPU modifications since that makes the most sense. Sometimes they try to use OS-level techniques. Here's a survey showing some of each:

https://thayer.dartmouth.edu/tr/reports/tr13-001.pdf

My first thought was to try to use 'containers' (cgroups) combined with the AMD secure memory extensions to achieve this type of isolation using as much off the shelf hardware as possible.

https://en.wikichip.org/wiki/x86/sme

https://www.kernel.org/doc/Documentation/x86/amd-memory-encr...

From the quick description it sounds like this provides a way of encrypting, per memory page, based on a symmetric key that is backed by some level of hardware encryption. It was not clear (in a quick read) how or where to specify the key by which an individual page is encrypted. That would be a critical component of comprehension with respect to identifying if this could be used to encipher individual processes and further isolate memory. It sounds like it might be possible to establish per-process memory isolation, which is probably the best level of security possible without resorting to entirely isolated hardware.

Per-process keys aren't really possible because memory can change process ownership (vmsplice) or be shared across processes (fork, page cache, memfd). It might be possible for pages marked MADV_DONTFORK

Additionally a per-process key does not help against spectre style attacks where you would trick the process into speculating on protected memory.

I've tried implementing a couple of (toy) password managers over the years and dealing with private keys is genuinely complicated. Even something as trivial as making sure that the memory gets zeroed correctly when you discard the key is trickier than it seems. Compilers these days are very good at detecting "dummy" memsets to memory that's never read afterwards and optimize them away. You have to use some dirty tricks to get the compiler to do what you want (copious amounts of volatile helps).

There's also the problem of making sure that your internal API is sound and doesn't copy the key around. In languages like C it's relatively easy to achieve (because copying a buffer in C normally requires some explicit code) but in higher level languages you might easily mistakenly pass the key object by copy instead of reference, leaving some duplicate of the sensitive information in some other location in RAM.

I think there really should be some kind of industry-standard multiplatform library meant to deal with secret keys that would implement all that behind the scenes and offer a simple API. It's simply too easy to make a dumb mistake when implementing these things, and you won't notice anything is wrong until somebody attempts to actually recover the key somehow.

> Compilers these days are very good at detecting "dummy" memsets to memory that's never read afterwards and optimize them away. You have to use some dirty tricks to get the compiler to do what you want (copious amounts of volatile helps).

Or you can use explicit_bzero(), which is designed for that use case.

Sure, although this is a non-standard and potentially non-portable extension. memset_s in Annex K is standard, but all of Annex K is optional, and like the rest of Annex K, it has an awful interface.

explicit_bzero or SecureZeroMemory are all insecure against those new sidechannel attacks we are talking about here. Only memset_s is.

You really need a mfence (full memory barrier), not just a compiler barrier, maybe even a clflush.

You might find this paper interesting in regard to memsets optimized away:

https://www.usenix.org/system/files/conference/usenixsecurit...

I agree with you on a standard platform. Id get a company like Galois to build it. They already have and open-sourced the necessary tooling from thrir prior contracts.

Yes, ALL private keys should be encrypted and stored elsewhere. The onion/knot should be unwound from some remote place, so you have to get access to a FEW places in order to get any private keys.

Now, in MY opinion, the whole idea of servers storing OR MANIPULATING unencrypted user-submitted information is what’s wrong with our current Internet culture. This needs to stop. It has massive social and economic effects and creates honeypots for hackers.

The sooner we make end-to-end encryption a basic expectation, the sooner we change our society to have a far more private and safe environment for everyone, and correct massive power imbalances: an organization or user should have power and data and connections and ability to SPAM others because people VOLUNTARILY gave them those things, not because they happened to build proprietary software and run it on infrastructure that is needed to operate the application!!!

> in fact one could argue that, the higher the quantity of memory chunks is, the harder it is to correctly decrypt the original private key; although to me this has the security through obscurity smell

I am not sure this is a bad place for security through obscurity. ASLR is similar. The thing is, here we are not using this is a primary mechanism. (that would be preventing RCE) Instead, this would be a secondary 'defense in depth' measure meant to make it harder to deploy an exploit after you have been compromised.

For such a second layer of defense in depth, obscurity is a decent option. Things go wrong if you start relying on your obscurity to do anything but slow down an attacker.

I feel like the should be a different name for that class of mitigations. Probabilistic security maybe? Security through obscurity for me is "we won't publish our algorithm". But that's not the same as ASLR or memory encryption which targets specific type of attack by adding probably-too-high randomness to the attack path.

AMD has some interesting work in this area.

https://github.com/qemu/qemu/blob/master/docs/amd-memory-enc...

RAM encryption is a feature in my password manager (java):

https://github.com/andy-goryachev/PasswordSafe/blob/master/s...

Curiously, RAM encryption, and its relative - clearing secrets from memory when not in use - cannot be added to the Bouncycastle library because they use java BigInteger (unless reflection is used of course).

How do you guarantee actual clearing of physical memory in a garbage collected VM?

by explicitly clearing it when an operation is finished.

example:

https://github.com/andy-goryachev/PasswordSafe/blob/master/s...

byte[] key = generateKey(); try { // sensitive operation } finally { Crypto.zero(key); }

once the operation finishes, the memory is cleared (and possibly subject to gc)

the problem is that some java classes, for instance BigInteger, are not designed for cryptographic operations - it's underlying arrays are not easily accessible (save for reflection).

That operation is almost certainly optimized out.

Would you elaborate please?

I believe he is referring to compiler optimisation which removes wasteful or extraneous operations. I don't write in Java though, so can't comment more than that.

The compiler will see that your operation does nothing and simply not do it.

What I mean was: do you know this to be the case for JVM 8 later? This is an interesting subject. Was there a study or a paper you can refer me to?

JVM 8 isn't an implementation, it's a specification.

Assuming you mean oracles implementation, it's likely that it is.

https://www.sjoerdlangkemper.nl/2016/05/22/should-passwords-... http://www.daemonology.net/blog/2014-09-04-how-to-zero-a-buf... https://man.openbsd.org/explicit_bzero.3

Thank you. I was hoping for a more definitive answer, may be a reference to an explicit test involving memory dump analysis.

In that code, after generateKey, during the sensitive operation, the system might need to do a garbage collection, at which point this array might have been copied to a different location in memory before your call to zero. You have to also "pin" (afaik this is the usual terminology for this) that array to a fixed location (which would then have to be a feature of that runtime and garbage collector) after allocating it but before generating the key.

So what you are saying is that JVM copying surviving object between generational pools might expose secrets in memory?

A very good point, thank you.

What could be done to mitigate this? Direct ByteBuffer per

https://stackoverflow.com/questions/5670862/bytebuffer-alloc...

?

In .NET you can pin memory in the GC heap, either using something like the fixed expression in C# or GCHandle. A quick Googleing did not turn up something usable directly by Java, except ByteBuffer.

Java also has no mfence or clflush support, so crypto.zero would never be secure. It might overwrite the key in the store buffer only, but not on the heap immediately, so prone to sidechannel attacks.

This is interesting. Could you explain this please?

You can't treat memory, even arrays, as singular memory locations in Java. Stuff moves around because collectors are cimpacting.. If you want to "write in place" in Java you need to either use a non-moving garbage collector, use sun.misc.Unsafe to read write memory directly, or mayyyyyybe take advantage of humongous allocations (arrays big enough that the JVM won't move them).

Direct ByteBuffer?

Yes, a direct byte buffer wrapped in class that ensures you can never copy the data somewhere else. That would also require cryptographic routines that can operate on bytebuffers or varhandles, which is quite an obstacle since many crypto libraries consume keys as byte[].

Ideally private keys should never touch the main CPU at all. You should use something like a secure enclave or a a key-dongle like Yubikey.

No. The CPU cache can easily be read out with sidechannel attacks via hyperthreading. It's the similar problem as unencrypted keys at the absolute location, which doesn't get cleared with explicit_bzero, which e.g. libsodium refused to fix. Hopefully crypto maintainers will get to their senses eventually.

https://github.com/jedisct1/libsodium/issues/802

libsodium has the sodium_mshield()/sodium_munshield(). For libhydrogen, storing a large prekey can be a problem, so it may only be implemented on some platforms.

If I were writing a cryptographic algorithm in C++, how would I ensure the CPU cache was used for private keys? Would it have to be written in a lower level language, or does there exist a library for C/C++?

Previous thread with lots of relevant information: https://news.ycombinator.com/item?id=8542405

Short answer is yes usually, and if not, you can always do inline asm. Of course, you're really, really, really not supposed to write your own crypto.

If everyone follows this advice, who will write the crypto code? If anything we need lot more people who are formally trained to write proper crypto code and find bugs in such code etc.

Let me qualify that. You're right, we do need a lot more people. But the answer is, don't write your own, write as part of a team. Ideally, a public and peer-reviewed project. The short answer is many people will work on it together, but don't write your own.