Hacker News Reader

Comment by ecnahc515 | Hacker News Reader

This has happened to basically every large web app company. Turn on debug logging in the app which logs HTTP request headers and likely doesn't strip out sensitive information. Easy mistake, and hopefully it wasn't like this since the beginning, and was maybe only found to be an issue for a subset of Robinhood's logs or something.

View on HN

This happened to Apple too, while logging API errors (CVE-2014-1317 - in this case it was hex-encoded).

Given the frequency that this occurs, I wonder if it'd be a good practice to use a sentinel value for a password during testing and grep logs for it.

And the hex encoded version of it.

base64, too, in all four alignments.

Maybe we should avoid sending passwords altogether. With some client-aide code, you could build a challenge-response system that eliminates this risk entirely.

Has there been any research into this or any companies that use it yet?

Amazon use it in their Cognito offering.

Chip-and-pin credit cards use it. 2FA/Fido/Yubikey use it.

Fido (and now w3c)'s WebAuthn come to mind

There's this thing: https://trykno.com/

As they're incredibly opaque about what they're doing, it's hard to tell whether they're part of the "get it wrong" crowd, but being incredibly opaque about what they're doing doesn't bode well.

I used this when I was doing web apps for small enterprises in early 2000's - I sent a random string in a hidden input in the login form and had some javascript concatenate the password and the random string and send me back the username, the random string and the hash of the concatenated string. I hope I got that right (not doing web apps any more since a long time ...).

You would want to use a token and the string and only accept the token to look up the string that you would only accept once and for a limited time.

Otherwise its vulnerable to replay attacks

Yes I think I was cashing the generated string server side for a while and checking if the string I received back had a matching entry in the cache, otherwise reject it.

Anyway, it's been a long time ago, and if I had to do it again today, i would rather not roll my own.

It is possible. But much easier to get wrong than to get right.

Specifically if you hash the password client side then the hash fundamentally becomes the password, and having that sit plaintext in logs is identical to the pre-hash password since both can re-played to authenticate.

I believe this is still a respected/secure version of what you describe:

https://en.wikipedia.org/wiki/Secure_Remote_Password_protoco...

No, having a client side hashed password sit in your logs is way better because it only introduces risk for your own service, instead of risk for every user who reuses their password on multiple services. It changes a compromise log from a widespread identify theft risk to a contained privilege escalation risk.

(Provided either you, or all the other services using this scheme, are sensible enough to salt with a string unique to their website such as a domain name)

just "hashing the password" is not a "challenge-response" protocol.

Thank you

if you can encrypt from the client side, you're always going to be able to reverse it too, its like storing the md5 of my plaintext password, just because it doesnt LOOK like my password in plaintext, it pretty much still is

> good practice to use a sentinel value for a password during testing and grep logs

Yes. I thought that was pretty common practice. We catch PII (not just password) log-leaks this way that verifying this way often enough that our developers have even learned[1] to check the searches before we yell at them when they try to take something leaky to prod.

If you don't automate detection of things like this you will make the same mistake again.

[1] Not being down on developers, I used to be one and still write a lot of code. Just noting a somewhat less than universal interest in compliance issues.

Any recommendations of tools/libraries that help with this sort of testing/analysis?

> Just noting a somewhat less than universal interest in compliance issues.

Is there a compliance document that requires this approach?

Or is it that the compliance states reasonable safety whilst handling PII and security credentials, and that it is down to the interpretation by each individual company as to what that means.

GDPR covers fines for logging sensitive things like passwords. There is additional separate certification needed if you're handling any payment details (i.e. Card numbers) yourself

I guess the struggle is a) what format its being logged in and b) knowing exactly where things are being logged/stored.

It would be even better practice to use type systems that don't allow the printing of values deemed to be sensitive.

Perhaps users could be asked to use their password-typed keyboard when entering passwords, and their address-typed keyboard for their addresses too?

Not sure if this is a joke, but iOS does distinguish between these.

"Secure" text inputs get the system keyboard, regardless of the user's custom installed keyboard.

So, if I understood correctly is that having a test info like username password etc and then grepping the logs for those known test info values and if it returns anything, then throw an error?

Yes. Ideally this would be an automated process, since it'd be tedious to grep every time you're messing around with logging, and you may sometimes forget to do it. If you're forwarding logs to something like ELK or Splunk, you can set up an alert / scheduled search to continuously check for those canary credentials every few minutes.

That's no excuse, especially for a financial services company like Robinhood. They have a duty to have process and people in place to prevent this exact type of issue.

plain text passwords in HTTP request headers? Wouldn't that be usually in POST body which almost never gets logged?

Headers often include session tokens, though

While careless and problematic, that is less problematic than logging passwords. The user probably reuses their password, so as soon as some developer can just look at logs and get a bunch of email/passwords, they probably now have access to other systems that they shouldn't.

That said, there are more than just logs to worry about. Is your service-to-service traffic encrypted? Then passwords are in the tcpdumps that you took to analyze a strange networking bug (you then copied the password to your workstation to look at the trace in wireshark). Do your programs segfault and core dump? Then passwords are probably in that core dump. Do you check the bounds of every memory read and write operation? Then passwords are probably in random variables (see: Heartbleed/Cloudbleed).

Sessions, though, are less scary. Sure, if you steal someone's cookie internally, you can impersonate that user on your own services. Nobody wants that, because it probably bypasses all the internal auditing systems; the auditing system can't tell that request apart from a legitimate user request. But, cookies can be revoked and have a shorter lifetime than a passwords, so it's not quite as bad.

Ultimately, it's all about limiting risk. If you have a database full of passwords, then someone compromising that database today gets all passwords ever. If you have a disk full of logs full of passwords, someone gets all the passwords that were used to log in within that log server's retention time period. If someone hacks in and starts tcpdumping your internal network and it's not encrypted, they only get passwords from users that log in while they're running tcpdump. If you encrypt network traffic, then someone only gets passwords that were in memory when your binary crashed. Nothing is perfect, but you can do more to add more security. Perfection is impossible, but not logging passwords is one step closer to perfection.

Session cookies should be locked to a TLS channel ID.

Then they are useless to anyone who steals them, since a TLS channel ID can't be recreated except with access to the TPM of the computer which initially created the cookie/session/TLS connection.

That requires support for token binding which google killed.

Can you post a bit more about how to do that? Your comment is a first result for my googling

Small enough POST bodies could still be worth logging to detect issues. Of course because of things like this it generally isn't a good idea, but there's still instances where it could be important to log POST bodies to eg. understand certain automated attacks.

I have, on a not insignificant number of occasions, used mod_dumpio in Apache to debug things. Which means I end up with all POST payloads in log files. Which has occasionally meant I ended up with plaintext passwords in log files... I am extremely careful when doing this. But I have seen cow orkers copy my debugging techniques and "forget" to switch off the dumpio logging - which has at least once been discovered when a prod server's log files started blowing out in size dramatically. With real user passwords in it... (Quite how mod_dumpio and it's config ended up on a prod server is another rant, which needs to be done over whisky, not in the office...)

https://en.wiktionary.org/wiki/cow-orker

This is why there is client side hashing. You still hash+salt on the server side, but the client side hashing prevents the worst kind of exposure.

What's the significant difference between sending a plaintext password and a sha256(plaintext password) to the server, except if the user reuses that password on other sites?

> except if the user reuses that password on other sites

Except you can't really except that.

But isn't it bad practice to send plain text passwords to the backend? Shouldn't you encrypt them in the client already?

"This has happened to basically every large web app company."

That is a seriously sad reflection on the poor quality of the software cranked out today.

Someone should be losing their job.

Sure, people should lose their job because someone's diagnosing a network problem, happens to run tcpdump on a server behind a TLS-terminating load balancer, and sees username/passwords. /s This literally happens all the time.

Depends, how did they get to the server (secure jumpbox)? Did anyone know it'd be a security related event when they were diagnosing the problem? Did the tech take precautions? In such a case, I don't think robinhood.com would need to notify users. They tcpdump would be handled securely, scrubbed properly, troubleshooting done, and that'd be the end of it.

If that isn't what happened, someone should be losing their job. The minute you think diagnosing network problems is a reason to ignore security to a degree that your users need to be notified you're incompetent.

If they used the info to hack accounts or other malicious activity, then yes, they should lose their job.

If they just left a log or capture file around somewhere, nope. Everyone makes mistakes.

"If they just left a log or capture file around somewhere, nope. Everyone makes mistakes."

I guess that's where you and I differ. There are certain mistakes that are tolerable. But ones that result in mass emailing to your clients that they need to change their passwords are not. I'd never heard of robinhood.com before, but now I have, and my impression is I'd never trust them because they have staff that makes mistakes with my secure information.

Think of how robinhood.com's employees feel about this. Management, and the people who are now watching the fallout rob the company of revenue.

Telling me that the person who made that mistake is gone might make me think differently.

The minute you start taking users name and passwords you're in the big boy world and need to treat it as such. Mistakes can be fatal, not just for your job, but your company.

I appreciate their transparency. It should make you trust them more because they're letting you know about the problem.

Some organizations would simply ignore this and move on. You'd never know it even happened.

I'll admit, that's a good point that made me go, "Hmmm."

Are you aware that Github, Twitter, and Facebook also logged passwords?

https://www.zdnet.com/article/github-says-bug-exposed-accoun...

https://arstechnica.com/information-technology/2018/05/twitt...

https://www.theverge.com/2019/3/21/18275837/facebook-plain-t...

You're proving my point. It's not taken seriously enough, even by huge companies with lots of resources.

Blameless post-mortems are a thing for a reason. Honesty about mistakes should make you trust an individual/company more not less.

Just my 2 c

Since we now put the user-id first, before the form goes to check how to even try to auth said user, we can just fetch a strong random salt from the server in it's response "just use a simple password, this is not a fancy account". Some nice, salted PBKDF shouldn't be a problem then.

Also, there are zero-knowledge authentication systems, but they are somewhat iterative due to their probabilistic nature. They are secure in the face of an imposter authentication service, as the password (or a hash, obviously) has to be stored in the real authentication service. But it is never transmitted after this initial store, so no amount of logging or posing as an evil imposter risks disclosure.

After having the risks explained to them and having received our professional recommendation that they add application-specific logging that had no means of accidentally logging PII, one of our customers (a major bank) still insisted that we implement full HTTP body logging in a proxy.

Nobody is going to lose their job over this because the practical consequences are that you have to send out an "our bad" email and offer a credit protection voucher that most customers won't actually use. There are no costs.

You’re both correct: someone “should” lose their job, but nobody will.

Not that that someone is necessarily the person who actually turned on this logging, it could be their supervisor who let it go live, or QA or… wherever the buck stops.

Reply on news.ycombinator.com