This doesn't mean they were stored in a database. "a readable format within our internal systems" could be log files if they didn't scrub passwords when logging requests.
That said, there are more than just logs to worry about. Is your service-to-service traffic encrypted? Then passwords are in the tcpdumps that you took to analyze a strange networking bug (you then copied the password to your workstation to look at the trace in wireshark). Do your programs segfault and core dump? Then passwords are probably in that core dump. Do you check the bounds of every memory read and write operation? Then passwords are probably in random variables (see: Heartbleed/Cloudbleed).
Sessions, though, are less scary. Sure, if you steal someone's cookie internally, you can impersonate that user on your own services. Nobody wants that, because it probably bypasses all the internal auditing systems; the auditing system can't tell that request apart from a legitimate user request. But, cookies can be revoked and have a shorter lifetime than a passwords, so it's not quite as bad.
Ultimately, it's all about limiting risk. If you have a database full of passwords, then someone compromising that database today gets all passwords ever. If you have a disk full of logs full of passwords, someone gets all the passwords that were used to log in within that log server's retention time period. If someone hacks in and starts tcpdumping your internal network and it's not encrypted, they only get passwords from users that log in while they're running tcpdump. If you encrypt network traffic, then someone only gets passwords that were in memory when your binary crashed. Nothing is perfect, but you can do more to add more security. Perfection is impossible, but not logging passwords is one step closer to perfection.
Then they are useless to anyone who steals them, since a TLS channel ID can't be recreated except with access to the TPM of the computer which initially created the cookie/session/TLS connection.
Given the frequency that this occurs, I wonder if it'd be a good practice to use a sentinel value for a password during testing and grep logs for it.
As they're incredibly opaque about what they're doing, it's hard to tell whether they're part of the "get it wrong" crowd, but being incredibly opaque about what they're doing doesn't bode well.
Otherwise its vulnerable to replay attacks
Anyway, it's been a long time ago, and if I had to do it again today, i would rather not roll my own.
Specifically if you hash the password client side then the hash fundamentally becomes the password, and having that sit plaintext in logs is identical to the pre-hash password since both can re-played to authenticate.
I believe this is still a respected/secure version of what you describe:
https://en.wikipedia.org/wiki/Secure_Remote_Password_protoco...
(Provided either you, or all the other services using this scheme, are sensible enough to salt with a string unique to their website such as a domain name)
Yes. I thought that was pretty common practice. We catch PII (not just password) log-leaks this way that verifying this way often enough that our developers have even learned[1] to check the searches before we yell at them when they try to take something leaky to prod.
If you don't automate detection of things like this you will make the same mistake again.
[1] Not being down on developers, I used to be one and still write a lot of code. Just noting a somewhat less than universal interest in compliance issues.
Is there a compliance document that requires this approach?
Or is it that the compliance states reasonable safety whilst handling PII and security credentials, and that it is down to the interpretation by each individual company as to what that means.
"Secure" text inputs get the system keyboard, regardless of the user's custom installed keyboard.
That is a seriously sad reflection on the poor quality of the software cranked out today.
Someone should be losing their job.
If that isn't what happened, someone should be losing their job. The minute you think diagnosing network problems is a reason to ignore security to a degree that your users need to be notified you're incompetent.
If they just left a log or capture file around somewhere, nope. Everyone makes mistakes.
I guess that's where you and I differ. There are certain mistakes that are tolerable. But ones that result in mass emailing to your clients that they need to change their passwords are not. I'd never heard of robinhood.com before, but now I have, and my impression is I'd never trust them because they have staff that makes mistakes with my secure information.
Think of how robinhood.com's employees feel about this. Management, and the people who are now watching the fallout rob the company of revenue.
Telling me that the person who made that mistake is gone might make me think differently.
The minute you start taking users name and passwords you're in the big boy world and need to treat it as such. Mistakes can be fatal, not just for your job, but your company.
Some organizations would simply ignore this and move on. You'd never know it even happened.
https://www.zdnet.com/article/github-says-bug-exposed-accoun...
https://arstechnica.com/information-technology/2018/05/twitt...
https://www.theverge.com/2019/3/21/18275837/facebook-plain-t...
Just my 2 c
Also, there are zero-knowledge authentication systems, but they are somewhat iterative due to their probabilistic nature. They are secure in the face of an imposter authentication service, as the password (or a hash, obviously) has to be stored in the real authentication service. But it is never transmitted after this initial store, so no amount of logging or posing as an evil imposter risks disclosure.
Nobody is going to lose their job over this because the practical consequences are that you have to send out an "our bad" email and offer a credit protection voucher that most customers won't actually use. There are no costs.
Not that that someone is necessarily the person who actually turned on this logging, it could be their supervisor who let it go live, or QA or… wherever the buck stops.
Except you can't really except that.
You must ALWAYS hash the password (as a salted iterated cryptographic hash) on the server. You can in ADDITION also hash the password on the client end, but by itself that's not enough.
The issue, as usual, comes down to "what is the attack you're trying to thwart"? If you hash on the client and not the server, then when (not if) the attacker manages to download the hashed password set, the attacker can create a modified client & send those hashes directly. If you don't hash on the server, it's exactly the same as storing passwords as clear text, because you're storing the data that an attacker can directly use to log in.
You can also hash the password on the client IN ADDITION to the server. In that case, you're hiding from the server the actual password you type in. That's an improvement if you're sharing a single password across many services; in this case the attack you're trying to thwart is to prevent an attacker from actively capturing the password & trying to reuse the password on other systems. However, a much better idea is to not share a single password anyway, so it's not such a great thing.
To be clear, it would have prevented these logged passwords from impacting other websites if the true cause of the Robin Hood password reset was a logging issue.
As a user, you can prevent this from happening either way by choosing strong, unique passwords for every service.
I've also wondered why this isn't more common.
[1] https://blog.cryptographyengineering.com/2018/10/19/lets-tal...
I wonder why PAKEs haven’t caught on?
This article (somewhat ironically) talks about it:
https://arstechnica.com/information-technology/2012/08/hacke...
Overview, Scheme_0: - You send your unique identifier (email, username, etc) and password to the server in plaintext. - The server looks up the password hashing scheme and salt associated with your identifier - The server checks that salt+password+hashing scheme produce the stored hash - Server kicks back proper response
In your scenario, Scheme_1: - You prehash the password locally and send your identifier + prehashed_password to the server - The server looks up the password hashing scheme and salt associated with your identifier - The server checks that salt+prehashed_password+hashing scheme produce the stored hash - Server kicks back proper response
The only difference is the extra hashing step. From a security perspective, there is no gain; storing the prehashed_password from Scheme_1 in plaintext (e.g. logs) is no different than storing the password from Scheme_0 in plaintext.
PAKEs are probably a good idea across the board.
As others have pointed out, if you go for complexity, there are better schemes. However, by avoiding the complexity you keep simpler client side code.
Especially because this is standard, whilst doing extra stuff would involve doing your own crypto code.
I guess you could install it as a Linux system which searches through all plaintext traversing the network AND filesystems of the entire infrastructure. Which might be a good business idea if you could pull it off.
https://www.bleepingcomputer.com/news/security/twitter-admit...
I remember on a bank's IVR I was auditing, required "_random 2 out of your 4 characters of a PIN_" (not your card's but dedicated to a call center support - that gave you full access to your accounts though) was being recorded/logged together with any phone key press (e.g. your customer ID). It was possible (if obtaining the logs of a few of a customer's calls) to reconstruct the PIN (e.g. call1#1 you press 1, 3, call#2 you press 1,2, call#3 you press 2,4). Those logs were scrubbed/dropped once a year, so for a frequent IVR user it was easy to reconstruct your IVR PIN. The folder containing the logs was a Windows shared folder with Everyone/Everyone (share/modify).
You should have seen the COO's face as I was describing how easily this could have been abused (likelihood), and how it could open a floodgate of serious problems for the bank (impact).
They've supported MFA for awhile, so simple enough to reset a password and make sure MFA is turned on.
I prefer when companies are proactive.
Security is a long term investment and it's easy to mess up. Taking responsibility and being transparent when you don't is the only way you're going to get it right.
There are some reverse proxies that are being sold as a way to mirror HTTP data to metric/analytics systems, which would bypass built-in filtering mechanisms which automatically remove passwords from logs (as Rails does). Metric/big data systems are an easy place for sensitive data leaks to happen. So your web developers could be getting security 100% right but some higher level systems integration with an analytics or ops software (requested by marketing or BI guys or whatever) could mess it up.
>>> When you set a password for your Robinhood account, we use an industry-standard process that prevents anyone at our company from reading it. Additionally, our policy is to not store usernames and passwords for any bank accounts that you link to your Robinhood account.
On Monday night, we discovered that some user credentials were unintentionally stored in a readable format within our internal systems. We wanted to let you know that your Robinhood password and linked bank account credentials may have been included. <<<
What level of incompetence is required to have accidentally logged bank account credentials?
This is not simple logging of login, oops, this is a very serious breach of basic security.
Why is the headline misleading? I mean, they say in black and white that they stored passwords in a readable format. The headline is completely factual.
If my password gets compromised, it's irrelevant where Robinhood stored it, just that they did so in plain text. That it was stored in text files, in a database, or handwritten on pink post-it notes doesn't make much difference.