Storing Your Data Securely: A Primer | Hacker News Reader

Storing Your Data Securely: A Primer

tinfoilsecurity.com

55 pointsShaneWilton41 comments

Not related to the article, but the source:

Having my email address on my blog is a vulnerability. Another vulnerability is named 'Rudimentary Scan', meaning 'we looked, and your site has no client-side vulnerabilities'. These issues mean my blog is 'borderline unsafe'. Wut.

Neither of these are vulnerabilities; to call them such does a disservice to your customer's understanding of technical language.

Sorry, perhaps it's a little unclear.

Rudimentary Scan means you ran a scan from our homepage, pre-verification, and we couldn't run our full suite of modules. All that is saying is that now since you're verified you should re-run a scan which will look for quite a bit more. We figured putting it there would be the best way of not giving a false sense of security from our 'rudimentary scan.' We make it 'borderline unsafe' so as to prevent that false sense of security. I can totally understand your confusion, and we should make this clearer. Ideas / feedback are always appreciated. :)

The email address is an informational vulnerability, which is dismissible.

Does that clarify things? (cofounder of Tinfoil)

If I were you, I would put the full scan at the end of the setup workflow (straight after verification), so that by the time the user gets to their report, you have full results for them.

Feedback appreciated. We're rethinking the whole flow, and we'll definitely try to make sure it's clearer. The intentions were good, I promise. :)

I'm not a security guru, but the most common scenario (even today, that big data and NoSQL/Graph dbs thrive) is a website running on a SQL (with some additional memcached/Redis or Mongo?).

Say I encrypt the entries... Since the database runs in local host, if the host is compromised and subsequently 'rooted', what exactly does database encryption offer?

The intruder will sniff/find the encryption key(s) since the key must be somewhere inside the application or in memory (don't know how, but I'm sure it's possible) in order to be able to decrypt data on the fly. The way I see it it's just added complexity for the admin with no gain.

Is there any way of defending an SQL database even when the host is compromised?

Simplest solution to fix this is to change "Since the database runs in local host". Otherwise, it depends on the extent of the compromise. If the attacker got root access, then all of the data is compromised. Note: you can minimize the effects of this compromise by doing one of a few things: storing the hash of data (e.g. with passwords) so that the original data is still (relatively) safe, or you could also have the data encrypted with another key (e.g. if it is a credit card number) that does not reside on the local machine. If the attacker has not gotten root access, you can use different users to separate what potential data the attacker has access to.

I think that it's probably safer to assume that if the attacker controls a local user account, escalating that to root privs "isn't that difficult." I see such comments all of the time here by people that should know about security.

Separating concerns be user account has its uses (e.g. sandboxing an app from accidentally screwing with files it doesn't need access to), but I don't think you necessarily want it as the linchpin of your encryption strategy.

That is not a fair assumption. The reason being that you know nothing about the type of vulnerability that they are exploiting. For example, if the site has a SQL injection attack, user separation in the database would prove quite useful.

> but I don't think you necessarily want it as the linchpin of your encryption strategy.

Its not about having one thing at "the linchpin". In fact, you should not rely on any singular thing, but rather separate out pieces of information so that no part has more than it needs. The test user should not have access to the production database and vice versa. In sum, you should use multiple things to defend yourself, not just one.

But if the database doesn't run on local host, the credentials to connect do it still have to.

Those credentials are only to a specific database user though. That user might not have access to credit card tables for instance. Therefore, even if the current exploit provides root access, those numbers might still be safe. Furthermore, it is a much simpler matter to revoke a users credentials than to deal with a machine which has been compromised in potentially numerous ways.

Not much. This is a genuinely hard problem. You want to keep data at rest encrypted, but data is restless, and it's hard to predict when each piece of it will wake up. So you end up keeping the decryption keys resident, accessible in some way to the running server. And, as you're intuiting, most things that compromise the server in some way pivot quickly to compromising the whole server, and thus the key.

The best answer I have to this problem is a variant of the HSM: the server stores ciphertext but not the key. To decrypt data, it's presented to another server, whose only purpose is to "seal" and "unseal" the data. That server:

* Doesn't run on shared hardware

* Has a totally different management process from the rest of the app servers

* Speaks only to authenticated, trusted hosts

* Is carefully monitored for usage anomalies

This isn't a panacea; in fact, its security value is surprisingly limited. But it does solve some actual problems.

A semi-interesting, completely off topic note on something I just learned after walking the earth for 28 years. The word "primer" as it is used here is actually pronounced primmer and not PRIMEr. Apparently the origins of both words are pretty unrelated.

OED accepts both pronunciations. Apparently "primmer" is the British way of pronouncing it.

> These functions belong to a family of functions known as key derivation functions: they use hashing to produce a digest which is suitable for use as a password hash.

KDFs are not for storing password hashes, they are for key derivation. This is a subtle point, the main thing is to use KDFs as KDFs (to derive keys that are then used to encrypt data). Details in the first link here (the second is about common mistakes when using scrypt):

- http://blog.ircmaxell.com/2014/03/why-i-dont-recommend-scryp... (bad title, he's referring to pass storage, not KDF usage)

- http://vnhacker.blogspot.com/2014/04/fairy-tales-in-password...

Another consideration is plausible deniability (PD) for situations where you're compelled to disclose your password(s). Our company writes Mac encryption software that specializes in this (and it uses scrypt). Here's a list of tools that offer PD (ours is called Espionage):

https://en.wikipedia.org/wiki/Deniable_encryption#Software

I compared Espionage's PD to TrueCrypt's on reddit:

http://www.reddit.com/r/security/comments/2b5icu/major_advan...

You're right, there's some ambiguous wording on my part. I'll be sure to make that section more precise right now. Looking back on that section, I should also be talking about work factors and such.

My apologies, I wasn't right. Please see this reply chain: https://news.ycombinator.com/item?id=8088208

You're right about parts of the post being unclear. I'll be rewriting that paragraph to be more precise regardless.

The first of these two source suggests that with typical settings, due to a quirk with the way scrypt works, it's less secure than bcrypt.

The second criticizes an API design with the scrypt code.

Neither makes a case that password-based KDFs are "not for storing password hashes".

Using a good password KDF as a password authenticator is a fine decision.

> The second criticizes an API design with the scrypt code.

Yes, I didn't claim otherwise.

> Neither makes a case that password-based KDFs are "not for storing password hashes".

The first one did quite explicitly for scrypt: "And that's why I don't recommend it for password storage."

> Using a good password KDF as a password authenticator is a fine decision.

Apparently not so for scrypt, and scrypt is the best KDF I'm aware of (heard something about yescrypt being better but haven't had a chance to look into it).

I don't see why a KDF is needed for password storage. What's wrong with SHA256 + random salt? If for some reason you want a slow hash function, use the salt MOD some constant to do several rounds of SHA256 (though I don't see why that would be necessary).

The last sentence of this comment is exactly why I called it out. If "for some reason" you want a slow hash function? You must have a slow hash function to store a password authenticator. The salt provides virtually no protection; it blocks a single corner-case exploit.

> You must have a slow hash function to store a password authenticator. The salt provides virtually no protection; it blocks a single corner-case exploit.

You're referring to simple dictionary attacks on a large database of passwords? Even bcrypt won't help much with a password from the top 10k list. I wouldn't say that stopping rainbow tables is "virtually no protection". Please explain what you're referring to.

Don't understand reason for the downvote. If I'm incorrect, an explanation would be appreciated.

SHA256+random salt goes at about 10e9 hashes per second on a workstation with a good GPU. That's enough to do a permuted dictionary attack that will get the overwhelming majority of passwords in a system.

[edit]

Does anyone know if the bitcoin ASICS have been repurposed as SHA256 crackers? md5 and sha-1 are more common in password storage, but if you use what the parent suggests, then it could be worse since there is so much specialized SHA256 hardware out there.

The bitcoin ASICs I'm aware of can't be used for cracking SHA256, because the function they compute is SHA256(SHA256(f(params)), where f is a function that stuffs some parameters into a bitcoin-specific block template before hashing.

The reason you are being downvoted is because you apparently dont agree with the entire purpose of password hashes. The reason you cant use a hash like the sha family is because they are too fast and prone to dictionary attacks. The solution is to slow them down. That is one of the primary functions of a password hash: to provide the ability to slow down the hash computation. This is usually done with a "work factor" which translates roughly to how many times you iterate with the hashing function. This repetition allows you to make the password hash to take arbitrarily long. With a work factor of 14 (2 ^ 14 rounds), bcrypt takes about a second to compute a hash on my machine. There are approximately 100k words in the english language and add one number and you are already at 1 million seconds to reverse a single hash. Compare this to the sha family where you take fractions of a second to hash a password.

> The reason you are being downvoted is because you apparently dont agree with the entire purpose of password hashes. The reason you cant use a hash like the sha family is because they are too fast and prone to dictionary attacks. The solution is to slow them down.

You misunderstand me. Please have a look at my reply to pbsd, I think it addresses your comments there too:

https://news.ycombinator.com/item?id=8087975

I did say there: "I stand corrected about KDFs not being OK for password storage".

I've read all your comments on this thread. I'm still unclear on why you think the problem of stretching a low-entropy secret into the high-entropy secret needed for strong cryptography is somehow different from the problem of safely storing a password so that disclosure of the authenticator doesn't directly compromise the password.

At this point, given the content on this thread, it seems that it's been explained clearly enough why salted SHAx is inadequate to that task. Do you still not understand why that is?

Until Windows finally got plugged into the Internet and the weak LANMAN hash became relevant to hackers, nobody used "rainbow tables" to break passwords (classic Unix crypt passwords aren't particularly vulnerable to them, since they're salted). But --- and this is crucial --- everyone cracked Unix password files. All of the original password cracking tools, from Crack through JtR, were designed to quickly crack salted hashes.

> At this point, given the content on this thread, it seems that it's been explained clearly enough why salted SHAx is inadequate to that task. Do you still not understand why that is?

Don't worry, I understand very well (this is not rocket science :P).

My problem, I think, was an inappropriate attitude toward the realities of stored password hashes on servers, in addition to having read that "scrypt is bad for password storage" (and having misunderstood the "badness" of it). I had read too many articles that treated HASH + SALT as "OK" for servers that stored user passwords, and I hadn't kept fresh in my mind the speed with which these passwords can be cracked today.

For example, take the article I linked to in my reply to pbsd: https://crackstation.net/hashing-security.htm (recently written).

This article, and many many others like it, are written by seemingly reputable people, and yet their attitude is that HASH + SALT is OK for servers. Even though it discusses bcrypt, etc., it frames it as though it's an "optional" hardening technique, and places HASH + SALT under the category of "proper hashing":

https://crackstation.net/hashing-security.htm#properhashing

I haven't seen them called out on it. I agree with you and the others in this thread, that it isn't good advice. If 10-char passwords can be protected better with KDFs, then they should be.

Every article you've read that suggests salted hashes are adequate stored password authenticators is wrong.

I don't know what the "security community" is, but if it has any relation to "the community of people who can speak with any authority on cryptography", let me ruefully assure you that it is much, much smaller than you think it is.

> Every article you've read that suggests salted hashes are adequate stored password authenticators is wrong.

Thanks for making that very clear. :)

Re "security community", sorry, I'd edited that part out before I saw your reply, and replaced it with a call for these posts to be called out for giving bad advice.

You mention yescrypt as an example of a good KDF, but its main design goal is to be a password hashing function!

The main difference between a password-based KDF and a password hashing function is that the KDF usually needs to support variable-length output. Otherwise they perform the same task: take a secret string of low-entropy s, and stretch the cost of breaking it to 2^(s + w) for some work factor w. In other words, a KDF is a superset of a password hashing function, and it's OK to use it as one.

> You mention yescrypt as an example of a good KDF, but its main design goal is to be a password hashing function!

I mentioned I'd heard it was a good KDF (not a good password storage function). I didn't know that was its main purpose. Thanks, will need to read more about it.

Still, there's marginal returns on how much w can be increased while keeping the KDF useful, right? There's only so much magic they can do for a weak password.

The article on scrypt being a poor candidate for password storage stuck out in my memory. I haven't seen or heard of the practice of strong hash function + random salt for password storage falling out of favor, so I didn't look into using other KDFs for that purpose.

EDIT: it does seem like a trade off. This article seems to have a good explanation of it: https://crackstation.net/hashing-security.htm

    If you use a key stretching hash in a web application, be aware that you
    will need extra computational resources to process large volumes of
    authentication requests, and that key stretching may make it easier to run
    a Denial of Service (DoS) attack on your website. I still recommend using
    key stretching, but with a lower iteration count. You should calculate the
    iteration count based on your computational resources and the expected
    maximum authentication request rate. The denial of service threat can be
    eliminated by making the user solve a CAPTCHA every time they log in.
    Always design your system so that the iteration count can be increased or
    decreased in the future.

I do stand corrected about KDFs not being OK for password storage. Seems like they can be overkill in some situations, but in general do a good job of protecting passwords.

Yes, realistically you can only add at most ~30 bits of security to a password. This does not save very poor passwords, but can save decent ones.

'Strong' means different things in different contexts. SHA-256, for example, is a strong hash function: it has good preimage and collision resistance. However, a function is evaluated by its weakest point: when given a low-entropy secret (which is not the assumption in a strong hash function) and a strong hash function, the easiest attack is to bruteforce it by sampling its distribution (e.g., the most common password patterns).

Since SHA-256 is fast, bruteforce is naturally also fast: the average time-to-break for a string of entropy s is 2^(s-1) times the cost of a SHA-256 evaluation. You can build on this by, say, iterating on SHA-256:

    def pwdhash(input, salt, w): 
        h = sha256(salt + input).digest()
        for i in range(2**w):
            h = sha256(h).digest()
        return h

But once you get into these ad hoc schemes to make the password hash less amenable to bruteforce, you're basically reinventing PBKDF2. scrypt (and bcrypt, to some extent) is like the above, but also uses random memory accesses on a large buffer to make things harder for specialized hardware (think GPUs, FPGAs, etc).

> Yes, realistically you can only add at most ~30 bits of security to a password.

This sentence makes no sense to me. tptacek, correct me if I'm wrong, but Im pretty sure you can increase w to be as large as you want (i.e. the keyspace is large enough that youll be waiting years for a singular hash to finish before you wrap the whole keyspace). The only problem is that you force legitimate users to wait as well.

Let make up an extreme example to illustrate, you set w so that a hash takes 1 years - 1 day to compute on the clients machine. Also, they change passwords every year. (Yes, this is absurd because it means that the client can only work one day a year). Now, lets assume that he uses one of five-hundred passwords. If the attacker has 100x the compute power of the client, he will only have a 20% chance of getting the correct password. And that is with only roughly 9 bits of entropy.

Take anything 'pbsd writes to the bank. The problem here is that adding enough cost to add significantly more difficulty than that to a password quickly takes you to places where the real-world cost is infeasible.

(He's not arguing against using KDF-like constructions to store password authenticators).

I've actually wanted to ask you about: https://news.ycombinator.com/item?id=8088253 but I cant reply yet. If that ever actually became an issue, could it be solved by instead having the client send (2 ^ w) - 1 rounds of the hash and having the server do the last round? My reasoning is that the client would have to brute force the full hash space if they wanted to reverse only the last round instead of trying to brute force the password.

one possible reason this doesn't work: A limit on the number of iterations for a password hash is the amount of latency the user is willing to accept, which is (approximately) constant regardless of whether those iterations are computed on the client or the server

> The only problem is that you force legitimate users to wait as well.

I think that's what he meant when he said "realistically". In the reply I posted the quoted text points out that for busy auth servers, it increases potential for DoS.

Zzz. This is a trope. Many of the largest app providers in the world use adaptive hashes to store passwords. All of them are DoS targets.

> Using a good password KDF as a password authenticator is a fine decision.

FWIW, I've seen several people claim that using PBKDF2 to store passwords is something only idiots do. They never explain why, of course; they just laugh at the people who don't know better. "lol PBKDF2 idiot." I don't know where this particular thought comes from, but it's out there.

The dumbest meme to emerge from generalist developers discovering adaptive hashing and KDFs is the idea that it matters a great deal which one you choose, and that any of the modern KDFs are a bad answer for password storage. It drives me nuts. Throw a dart at a dartboard labeled "scrypt", "bcrypt", and "PBKDF2". You're fine no matter where it lands.

Nerds can't resist any opportunity to turn something into an editor war.

Reply on news.ycombinator.com