Hacker News Reader

Comment by hak8or | Hacker News Reader

The diffirence is intent and/or gross negligence. Storing passwords in plain text in a database requires a gross lack of security mentality in both design stages and implementation changes. It is also drilled into people's brains constantly about how bad of an idea it is. To put it simply, it cannot happen accidentally.

Logging like this can easily be attributed to an accident. The person who implemented this logging should get hit with some repercussions because he surely tested the logging and must have seen the passwords when glancing by eye. But other than that, this was clearly a minor oversight.

View on HN

Yes. Consider this not-unlikely scenario: The people who implemented the logging did it as a feature of a generic API proxy (there’s no way Facebook implements logging separately for each of their bazillion services!), and no doubt put in a provison for masking sensitive data. They tested it and it worked fine.

Then some devs miles and years away didn’t use that feature properly and accidentally failed to not log passwords in an incoming request. They may not even have been looking at those request logs because that’s not the request they were testing.

Then that feature went into production and this oversight was magnified millions of times.

At large scale you don’t just tail the production log firehose and look for stuff. You have to search for specifics to find anything st all. So if nobody was debugging this thing in production it’s quite plausible nobody saw the passwords in the log.

One way to catch this sort of thing is sentinel data — in this case, have a unique value for a test account’s password and test every service with it, then search everywhere you can think of for that value.

So here is the thing: It was presumably relatively easy for you to come up with that scenario, which you called "not-unlikely". Then what you do is you put that scenario into your risk analysis when you're designing the authentication architecture, and figure out mitigations to make sure that particular mistake becomes (very) unlikely.

The notion that "it could easily happen" that is being brought up throughout this thread should really only suggests that people aren't doing even rudimentary security assessments (or, hopefully, they're not working with security sensitive software).

If you can't solve it technically, you solve it through processes and training. Same goes for any other industry -- if a construction worker said that it's just one bad morning away from dropping a two tonne girder on a playground, we would never accept that. Or a pilot crashing an airliner into the waiting hall when they're supposed to land. Somehow it seems that large parts of the software industry simply hasn't reached the level of maturity we expect from pretty much all other industries.

Facebook is an enormous company. They should be able to have entire departments working on these topics. It's not a one-person hobby project we're talking about.

> The notion that "it could easily happen" that is being brought up throughout this thread should really only suggests that people aren't doing even rudimentary security assessments

Precisely.

I'm sure they did figure out mitigations. They failed. Things fail. Two airliners just failed rather spectacularly, and that's the very industry you're benchmarking against.

>Somehow it seems that large parts of the software industry simply hasn't reached the level of maturity we expect from pretty much all other industries.

True, but that's a rather broad brush — in terms of actual risk of damages there is nowhere near an equivalence between "airliner crashing into waiting hall" and "logging some plaintext passwords".

Of course the culture, priorities, and domain are also very different between social network engineering and airliner engineering, which is by the way one reason Facebook could grow from nothing to mind-bogglingly gigantic in a decade, while it takes a decade to get just one new airliner into production.

The point I was making by comparing to a pilot, which I realise I could have expressed a lot more clearly, is that it is perfectly possible to mitigate risks through proper training and procedures even if it's not possible technically. (I.e. all it takes for a plane to crash is to turn the flight controls a few centimetres in the wrong way at the wrong time, yet it almost never happens.)

Of course things fail and people screw up. What I don't agree with are arguments along the lines of this just being a slight oversight, and that those can easily happen. It should require serious failure on multiple levels for anything like this to happen at that scale, if they are implementing things properly, not minor oversight.

Exactly — my scenario was an example of how failures at multiple levels could have caused this to happen. My "not-unlikely" is meant retroactively — now that it's happened, what's a not-unlikely explanation for how it was allowed to happen in a company the size of Facebook?

I didn't intend to imply it was a "slight oversight" — it's clearly a significant oversight — but there are people saying it's obviously gross negligence because how could this ever happen in a company that wasn't completely incompetent, etc. No, terrible accidents can and do happen even in companies that are trying hard to do a good job. Just like when a 737 crashes, you shouldn't assume Boeing is totally incompetent, but rather that several things must have gone wrong at once.

How is this in any way better or more excusable? Calling it a 'minor oversight' could also apply to the DB being in plaintext.

Storing passwords in a plaintext DB heppes in the same manner-a dev is lazy or ignorant of the security reprocussions. Which is what happened here; barring evidence this was done maliciously we can assume that this was accidental. But that doesn't make it any more excusable, since it should be clear that logs can also contain sensitive data that needs to be protected/anonymized.

Considering how selective FB is for hiring, I would hope we could expect a higher standard.

Not to mention 200-600 million users had their passwords exposed for many years. That must be a massive trove of log files.

I see this differently. If someone stores passwords in plaintext in the database, they’re idiots that don’t know any better.

If someone logs your password in plaintext but has it encrypted in the database that’s grossly negligent.

You assume that storing passwords in plaintext is intent as opposed to gross negligence. In many past instances it’s been intentional and gross negligence because the people making all the design and implementation choices were not knowledgeable about best practices.

Reply on news.ycombinator.com