Twitter urges users to change passwords after computer 'glitch'
reuters.com
reuters.com
"Due to a bug, passwords were written to an internal log before completing the hashing process. We found this error ourselves, removed the passwords, and are implementing plans to prevent this bug from happening again."
Exact same thing that github did just recently.
Their response is acceptable and textbook. Doesn't really seem like the appropriate place to wage the battle.
Now.
(We caught ourselves doing it 4-5 months back, and went through _everything_ checking... Only random accident that brought it to the attention of anyone who bothered to question it too... Two separate instances by different devs of 'if (DEBUG_LEVEL = 3){ }' instead of == 3 - both missed by code reviews too...)
if ( 3 = DEBUGLEVEL )
(I'm not even sure how they'd ended up with a Grails configuration that'd let them do this anyway...)
if ( level = DEBUGLEVEL )
As an experienced programmer I have written if-statements so many times in life that I never ever, even by mistake, type:
if (a = b)
if (a == b)
if ((a = b) == c)Like other people are saying - the toolchain should have caught this. And it should have, I don't remember how it'd been disabled...
If you must write code, errors follow, and “defence in depth” is applicable. Use an editor that serves you well, use compiler flags, use your linter, and consider Yoda Notation, which catches classes of errors, but yes, not every error.
if (myVar) {
//
}
The accidental assignment is much less common due to the way equality is tested in Java.
Also, `null` comparisons being assigned will fail to compile (assuming var is a String here):
TestApp.java:6: error: incompatible types: String cannot be converted to boolean
if (var = null) {1) Outside of initialization, assignment is done with the `<-` operator, so you're only potentially confused in the opposite direction (assignments incorrectly being boolean comparisons).
2) Return types and inputs are strictly validated so an accidental assignment (returning a `void`/`unit` type), would not compile as the function expects a bool.
3) Immutable by default, so even if #1 and #2 were somehow compromised the compiler would still halt and complain that you were trying to write to something unwriteable
Any of the above would terminally prevent compilation, much less hitting a code review or getting into production... Correctness from the ground up prevents whole categories of bugs at the cost of enforcing coding discipline :)
if (env('LOG_LEVEL') = 3) {}
Would throw and everything would be ok. Otherwise, use constants.
Also, lint against assignment in if/while conditions. If you want to assign in those conditions, disable linting for the line and make it explicit.
But ultimately, even if you do find it harder to parse (for whatever reason(s)) that would only be a training thing. After a few days / weeks of writing your comparisons like that I'm sure you'll find is more jarring to read it the other way around. Like all arguments regarding coding styles, what makes the most difference is simply what you're used to reading and writing rather than actual code layout. (I say this as someone who's programmed in well over a dozen different languages over something like 30 years - you just get used to reading different coding styles after a few weeks of using it)
Often when I glance over code to understand what it is doing I don't really care about values. When scanning from left to right it is easier when the left side contains the variable names.
Also I just find it unnatural if I read it out loud. It is called Yoda for a reason.
If you've ever spent more than 5 minutes listening to arguments and counterarguments regarding Python whitespace vs C-style braces - or whether the C-style brace should append your statement for sit on its own line - then you'd quickly see that all these arguments about coding styles are really just personal preference based on what that particular developer is most used to (or pure aesthetics on what looks prettiest - that that's just a different angle of the same debate). Ultimately you were trained to read
if (variable == value)
if (value == variable)
Furthermore any decent modern compiler will warn you and ask to add an extra set of parens around assignments in conditions so I don't really think it's worth it anymore. And of course it won't save you if you're comparing two variables (while the warning will).
You will have a better, more reliable, and safer codebase once you clean up the legacy mess and turn this on..
Plus you get out of the habit of not reading output from the compiler because there are so many warnings...
> if a = 3:
^
SyntaxError: invalid syntaxOf course a consequence of that is that you can't chained affectations (a = b = c) but it's probably a good compromise.
But luckily it's something we cover in code reviews and the logging mechanism has a "sensitive data filter" that defaults to on (and has an alert on it being off in production.)
It only takes one debug statement leaking to prod - it has to be a process, not an event.
Any company that deals with credit card data has to be very very sure that no sensitive data is written in clear anywhere. Even while in memory, the data needs to be hashed and the cleartext data erased as soon as possible. Per what I have heard from friends and colleagues, the other popular companies like Amazon, Twitter, Netflix, etc. also have similar processes.
Checking logs for sensitive data is a routine test when given access atleast.
Being given that access is disappointingly not routine though.
Do you enable debug logging in production? In our setup we log at info and above by default, but then have a config setting that lets us switch to debug logging on the fly (without a service restart).
This keeps our log volume down, while letting us troubleshoot when we need it. This also gives us an isolated time of increased logging that can be specifically audited for sensitive information.
Create a user with an extremely unusual password and create a script that logs them in once an hour. Use another script to grep the logs for this unusual password, and if it appears fire an alert.
Security reviews are important but we should be able to automate detection of basic security failures like this.
But you can do fancy cryptographic things where the server never sees the password and it's still secure. like the entire field of public key cryptography, diffie-hellman key exchange, etc.
Someone in another part of this thread mentioned the "Web Authentication API" for browsers, which I'm not familiar with, but is possibly trying to approach this?
It ties in with the credential management API (A way to have the browser store login credentials for a site, a much less heuristic based approach than autocomplete on forms) and basic principle is generate a key pair, pass back public key to be sent to server during registration. On login generate a challenge value for the client to sign. I don't think iirc the JS code ever sees the private key, only the browser sees it.
edit: considering someone eavesdrops on the connection, otherwise that's a whole different kind of vulnerability
register: send (username, user_salt, HMAC(user_salt, pwd))
login: send (username). retrieve user_salt. retrieve a server_salt generated randomly. send HMAC(server_salt, HMAC(user_salt, pwd))
But now your password is effectively just HMAC(user_salt, pwd), and the server has to store it in plaintext to be able to verify. Since plaintext passwords in the db are bad, this solution doesn't sound too attractive, unless you were suggesting something else.
--
Server stores hashed-password, hash-salt, and random-salt.
Server sends hash-salt, and random-salt to client.
Client uses user password and hash-salt to generate hashed-password.
Client hashes hashed-password using random-salt.
Client sends hashed-hashed-password to server.
Server grabs stored hashed-password and hashes used stored random-salt to check for match against client's hashed-hashed-password.
--
So the only thing this actually does is not share the text of the password that the user typed to the server. But at a technical level, now the hashed-password is the new "password".
Let's say the database is compromised. The attacker has the hashed-password. They make a login request to fetch the random-salt, hash their stolen hashed-password with it and send that to the server. Owned.
Along with being more complicated with no real gain, this also takes the hashing away from the server-side, which is a big negative, as the time that it takes to hash a password is a control method used to mitigate attacks.
Just send the plain-text password over HTTPS and hash it the moment it hits the server. There's no issue with this technique (as long as it's not logged!)
> Alternatively a timestamp would be just as good.
I don't see how that would work at all.
I also don't see the need to go any further in detail about how this scheme will not be better than the current best practices.
Never. Roll. Your. Own. Crypto. https://security.stackexchange.com/questions/18197/why-shoul...
Incidentally, I really resent how it's impossible to have a discussion of anything at all related to cryptography on HN without somebody bringing up the "never roll your own crypto" dogma.
If the ideas being proposed are bad, please point out why, don't just imply that everyone except you is too stupid to understand.
Edit:
I just reread your comment above and you did a perfectly good job of explaining why it's a bad idea, I must have misunderstood first time round: it's a bad idea because now the login credentials get compromised in a database leak instead of a MITM, which is both more common in practice and affects more users at once.
Sorry for saying you didn't explain why it is a bad idea.
So now _this_ is effectively just "the password", that needs to be protected, even though you're storing it server side.
If an attacker has it, they can go through the protocol and auth -- I think, right? So you prob shouldn't be storing it in the db.
All you're doing is shuffling around what "the password" that needs to be protected is, still just a variation of the original attempt in top comment in this thread.
The reason we store hashed passwords in the db instead of the password itself is of course because the hashed password itself is not enough to successfully complete the "auth protocol", without knowing the original password. So it means a database breach does not actually expose info that could be used to successfully complete an auth. (unless they can reverse the hash).
I _think_ in your "protocol" the "original" password actually becomes irelevant, the "salted hashed password with it's salt" is all you need, so now _this_ is the thing you've got to protect, but you're storing it in the db, so now we don't have the benefits of not storing the password in the db that we were hashing passwords in the first place for!
I guess your protocol protects against an eavesdropper better, but we generally just count on https/ssl for that, that's not what password hashing is for in the first place of course. Which is what the OP is about, that _plaintext_ rather than hashed passwords ended up stored and visible, when they never should have been either.
Cryto protocols are hard. We're unlikely to come up with a successful new one.
This is exactly the same: the seeded hash is the password.
Also, hashed passwords shouldn't be logged either.
No [0,1...n]. Note that these articles are about encryption, but the arguments against javascript encryption apply to hashing as well.
Also consider that no one logs this stuff accidentally to begin with. If the entity controlling the server and writing the code wants to log the passwords, they can rewrite their own javascript just as well as they can whatever is on the backend. There's nothing to be done about people undermining their own code.
[0]https://www.nccgroup.trust/us/about-us/newsroom-and-events/b...
It's possible. You create an object called Foo (possibly a serialized data like a protobuf, but any object), and you recursively dump the whole thing to the debug log. Then you realize, oh, when I access a Foo, sometimes I need this one field out of the User object (like their first name), so I'll just add a copy of User within Foo. You don't consider that the User object also contains the password as one of its members. Boom, you are now accidentally logging passwords.
Creating a User object that holds a password (much less a password in plaintext) seems next level stupid to begin with, but fair enough, I guess it could happen.
It can happen if requests are logged in middleware, and the endpoint developer doesn't know about it. It's still an extremely rookie mistake though, regardless of whether it was done accidentally or on purpose.
Maybe there could be a standard way to signal the beginning and end of a password string so logging software can redact that part.
My guess is that this isn't popular because of the added client side complexity.
I'm also curious if anyone has considered or implemented this idea.
Although I think this still improves the situation if the password is reused. I.E. I can't use the logged hashed password on other sites.
You could hash it twice (once on the server once on the client) I suppose, but I'm not entirely sure what the benefit of that would be.
I'm imagining we have a system where a client signs, and timestamps, a hash that's sent meaning old hashes wouldn't be accepted and reducing hash "replay" possibilities ... but now I'm amateurishly trying to design a crypto scheme ... never a good idea.
How would the server even verify the hash, then?
Ideally all users would change their passwords to something completely different in the event of a leak. But realistically this just doesn't happen -- some users refuse to change their passwords, and others just change one character. If only the client-side hash is leaked rather than the raw password, you can greatly mitigate the damage by just changing the salt at the next login.
In case we are talking about multi-tier applications where probably LDAP or AD is used to store the credentials then the back end is the one responsible for doing the hashing.
I’m not seeing how it would be a challenge in a uuid based environment, unless there’s a nuanced detail I’m missing.
Have an automated system send periodic login requests (or any other requests which contain sensitive information that shouldn’t be logged) for this account, and have another system which searches log files for the password.
If it’s ever found, you know something is leaking.
A better advice is to delete accounts you don't use. If not possible (illegal in EU now) scramble private data and the password.
Download the databases yourself and check them locally.
Changing passwords regularly also limits the damage.
My (limited) experience makes me think that cleartext passwords are somehow hard coded to be logged, perhaps through error logging or a feature that’s intended for testing during development.
I personally would not code a backend that allows passwords (or any sensitive strings) to be logged in any shape or form in production, so it seems a little weird to me that this mistake is considered a “bug” instead of a very careless mistake. Am I missing something?
EDIT: Thank you very much in advance!
So sure you don't want to log everything in Prod, but maybe you do in Dev. In that case, a bug would be to push the dev logging configuration to Prod. Oops.
If you have the clear text password at any point in your codebase, then there is no full-proof way to prevent to log it unintentionally as the result of a bug. You just have to be extra-careful ( code review, minimal amount code manipulating it, prod-like testing environment with log scanner, ...)
It turns out that this is non-trivial - when censoring how do you indicate that something was changed, while keeping the output to a minimum? blank/"null" was rejected because it would mask other problems, and "* THIS FIELD HAS BEEN REDACTED DUE TO SENSITIVE INFORMATION *" was rejected for being "too long". Currently we use "XXXXX", which has caused some intern head scratching but is otherwise fine.
If it's just your authentication system hashes that are compromised, the damage can be contained.
There are ways with downsides to mitigate the risk logging requests.
HMAC with time component will render the data useless before long. Essentially OTP. Downside client time needs to be accurate.
Negotiate a shared key ala NTLM. Downside more round trips; essentially establishing encrypted transport inside encrypted transport (https).
The argument was essentially that if Twitter had instead chosen a language that was more natively inclined toward scalability, they would have necessarily hired 10x as many engineers and they would not have succeeded at building the product that people use to tell each other what bar they are at, simply, which ultimately was the braindead simple thing (that you can probably scale just fine in any language) which drove their success... it wasn't any great technological feat that made Twitter successful, it was pretty much just the "Bro" app that people loved.
(The talk was called "Rails Doesn't Scale" and will be available soon, but RailsConf2018 evidently hasn't posted any of the videos yet.)
Also they say they found no evidence of anyone stealing these passwords, but I wouldn't be surprised if some companies decide not to look too hard just so they can later say "they found no evidence of such an act."
How does one do that?
https://www.mac4n6.com/blog/2018/3/30/omg-seriously-apfs-enc...
>Write passwords to a log
Security level - Twitter.
1. Make a list of all sensitive information that the test users in your test environment will be giving to your application.
As part of your test procedure, search all logs for that information. This can be as simple as having a text file with all the sensitive information, and doing a 'grep -F -f sensitive.txt' on all your log files.
2. If you are working in some sort of object oriented language, make separate classes for sensitive data, such as a class Password to hold passwords, a class CardNum to hold credit card numbers, and so on.
Make the method that whatever you use for logging uses to convert objects to strings return a placeholder string for these objects, such as "Password", "Credit Card Number", and so on.
1. (most pertinent in this discussion) The logging framework can redact fields before emitting log entries. 2. Endpoints can redact the data unless the client explicitly requests (and is allowed to receive) unredacted data. 3. Serialization mechanisms (e.g. Gson) can be configured to redact data before serializing. (Again, probably can't always do this, but can make that the default for safety.)
It's also very straightforward to hook up as a Java annotation that does the same things.
This is a hard notion to accept because it has to be built into the organization. No silver bullet, no consultant, no five-step methodology. Just rigorous software engineering up and down the stack, and care, and attention to detail.
This assumes that passwords and credit card numbers are more sensitive than session cookies or tokens, which is usually true because people tend to share passwords across sites and definitely share credit card numbers across sites. So the risk of logging cookies/tokens is much lower. Also, you can revoke all cookies and token much more easily than you can force everyone to change passwords or credit card numbers.
If rather than a naked string you have a Password class with literally no way to extract the plain text password then you can be positive that any code that uses it will never accidentally log the password.
In contrast, if you rely on microservice tokenization you can still accidentally log your password before your tokenization happens, just like people are accidentally logging passwords before they are bcrypted.
Both proposals have a problem of logging raw requests or raw service calls (outside of your object model).
Where plausible it would probably be best for microservice calls to only take already bcrypted passwords, and error out if it detects one that isn't, so there is zero chance of accidentally logging a plain text password when calling a microservice.
Again, an actual object (e.g. HashedPassword) shines here, because your code can automatically detect bad values the instant it hits your object model, and refuse to properly log or give access to anything that looks like it isn't already hashed.
I think the risk of logging raw passwords in the auth service model is lower because logging in a password-specific microservice is an intuitively dangerous thing to do, so both code authors and code reviewers will pay heightened attention to it. Meanwhile, "log all requests" is a common thing to want to do and will raise fewer alarms in a primarily-business-logic service. (In fact, another usually reasonable thing to do is "log all requests that don't parse properly and return 500...")
Besides securely managing passwords, you can also use a password manager to secure your digital legacy. 1Password has a feature where you can print out "emergency kit" sheets that has the information required to access your password vault. I printed out two of these sheets and gave them to trusted family members in sealed envelopes. In the event that I become incapacitated, they will be able to access my accounts.
It is not open source but the vault format is open.
I've no affiliation, just a satisfied customer.
I set it up to need both a private key and password to unlock my password DB. The private key moves around on a thumbdrive only (never in Dropbox).
I like that the only parts of this system I have to trust are open source.
I just wish there was more seamless support for apps to use 1Password to paste in passwords. There are still sites that prevent pasting into password fields!
https://www.passwordstore.org/
It uses a similar philosophy of encrypting plain text files and you can sync them how you wish. It might do some of the 'heavy lifting' for you.
1Password can also store other information besides passwords such as credit cards, software license numbers, passport numbers, etc. There is also a secure notes feature for storing arbitrary text.
The other password manager that I tried before 1Password is Lastpass. I ended up choosing 1Password since I think it's better designed and overall feels slicker. The /r/lastpass subreddit is littered with complaints about broken updates and bugs...
It will allow you to organize the passwords in a hierarchical way with folders (banks, administration, forums, whatever), and set icons.
It will also keep the date of the last time you modified it. Sometimes this can be useful to know if you are impacted by a breach revealed after the fact. You can also make passwords expire if you like.
You can also add extra data in a way that doesn't clutter the main view. This can be interesting when credentials are more than login/password. For example you could add a PIN there. For my car radio there is a code to enter to make it work after the battery dies, I added the entire procedure to the extra data as I always forget it and it's not intuitive.
I just checked, I have 957 passwords in my KeePass.
Every once in a while I hear someone explain their system for this (and I used to use a simpler scheme), and I can think of arguments about why it won't work for long, but I'd be happy to update my internal monolog from actual evidence.
I've got about 1K passwords in 1Password. I generally rotate them when they're 1-3 years old, depending on the threat model, cost of compromise, and on what I'm using password review to procrastinate.
Typically they don’t remind you when you are logging in that *we made you use 8 character passwords but you can’t use some special characters” or whatever. So you have to have some way of remembering what crazy password rules they had 3 years ago when you registered...
I really wish websites would support use of client side TLS certificates as part of the authentication process. Combining that with a username and password would give you two-factor authentication.
You could imagine a scheme where you give a user a certificate with a random subject, and you have a server-side map of random string to user account (so leaking that map isn't the end of the world like leaking passwords, it merely reintroduces the privacy leak above). I recall some proposals for that several years ago. Today, Web Authentication effectively does something effectively equivalent, as does U2F, although they don't involve TLS client certificates specifically.
If it matches the username I have on a website like reddit or HN, then is it really a privacy issue? Anyone, regardless of whether they're logged in or not, can see posts I've made under my username. Though what you say can be an issue for websites where privacy from other users is expected (e.g. banks).
> Today, Web Authentication effectively does something effectively equivalent, as does U2F
Both of those seem to rely on HTTP, while TLS could work with any application level protocol.
Another nice thing is that TLS 1.3 servers can send a CertificateRequest asking for a particular _type_ of certificate, so (if that's ever used in anger) it lets us have clients that don't need to waste the user's time when they don't actually have a suitable certificate anyway. In earlier versions servers could only hint about which CAs they trust, not anything else.
Do you only use Apple devices? Despite spending most of my time on a laptop with macOS, I also have a gaming PC with Windows, a home server with Debian, a mobile device with Android, and a tablet with iOS. It's nice to have a bit of flexibility available.
If you use an alternative browser such as Firefox you lose access to the built-in integration.
I think their SaaS offering has vault sharing for friends and family, which isn't available through iCloud Keychain.
They provide additional security audit features, such as vulnerability tracking. Quite relevant: I just opened the app and Watchtower had a vulnerability alert notifying me to update my password on Twitter.
It supports One-Time Password, which can occasionally be convenient.
Other kinds of item are supported as well, such as credit cards, bank accounts, software licenses, identities, and secure notes. No more having to grab for my wallet when I need to input my credit card or driver's license info. No more having to search for a checkbook to find my bank account number.
I use KeePassXC [1] with Syncthing [2] to synchronize my passwords between machines. No third-party!
[1] : https://keepassxc.org/
[2] : https://syncthing.net/
No matter what you do if the computer you are using isn't trustworthy you're losing.
[1] https://keepassxc.org/docs/#faq-platform-mobile
[2] https://itunes.apple.com/us/app/minikeepass/id451661808?mt=8
[3] https://itunes.apple.com/us/app/keepass-touch/id966759076?mt...
Login by email should really become a thing. There's just no reason to store passwords for most sites where you can just stay logged in indefinitely. On rare occasion you need your login cookie refreshed, just send a new link to your email. The burden of remembering a thousand secure and unique passwords dissolves immediately.
Second, sometimes sites mandate that you change your password. Or have rules that are incompatible with that algorithm. And then you need to start remembering exceptions to your algorithm, at which point you're back where you started.
I gave up on it for the same reason. Having to remember exceptions, plus when you change your password, you have to change it everywhere, which is annoying because you can't remember every active account.
"We are sharing this information to help people make an informed decision about their account security. We didn’t have to, but believe it’s the right thing to do."
The "we didn't have to" is a little jarring given the scale of this.
I know I wouldn't trust my password with the number of people that have easy access to logs at other large(ish) tech companies.
I really can't imagine why "we didn't have to" was included in that tweet, at all. What other flaps like this have occurred that exposed my creds or personal data to large numbers of employees, that they didn't have and didn't choose to tell us about?
Ideally, you'd only need to read raw logs tied to a test account, or, maybe your own personal accounts.
Stack traces and exceptions and the like can be anonymized and collated.
Maybe I miss the point behind this comparison? I guess I'd understand more if I thought the number of folks with node access and log access were in the same magnitude at Twitter, or if the TLS stack persisted data over time.
Nothing is known to have ever left Twitter's servers.
FTFY.
The fact it didn’t leave Twitter doesn’t mean everything is good. There are still a LOT of people who may have had some kind of access to this data.
The fact that they undeleted it is strong evidence that he didn't have discretion in how he performed his job, and thus was actually an employee and not a contractor.
How come? I interpreted it to mean that no regulations required this, but they chose to anyway. Which is true.
I think the tone matters. Particularly for matters like this.
To my ear (and many who replied to that Tweet) this reads as "we've decided to do you a favor." Which is very much not the case.
Compare that to Github who just yesterday went through the exact same problem, except they're requiring users change their password. Their CTO didn't make some 'hey you should THANK US" claim.
> During the course of regular auditing, GitHub discovered that a recently introduced bug exposed a small number of users’ passwords to our internal logging system, including yours. We have corrected this, but you'll need to reset your password to regain access to your account.
> GitHub stores user passwords with secure cryptographic hashes (bcrypt). However, this recently introduced bug resulted in our secure internal logs recording plaintext user passwords when users initiated a password reset. Rest assured, these passwords were not accessible to the public or other GitHub users at any time. Additionally, they were not accessible to the majority of GitHub staff and we have determined that it is very unlikely that any GitHub staff accessed these logs. GitHub does not intentionally store passwords in plaintext format. Instead, we use modern cryptographic methods to ensure passwords are stored securely in production. To note, GitHub has not been hacked or compromised in any way.
Personally, I'd probably just say "We felt that we had to disclose this to protect the interests of our users" (and not acknowledge that they might not, or that they had to make a decision to do so), or just say what they _are_ doing ("We are disclosing this in order to protect our users") and avoid the possibility that it is misinterpreted. I don't think there is anything to be gained by saying that they might not have done this.
I chose to use a manager anyway, with the logic that if my system was compromised I was owned either way; memorizing passwords provided very little, if any, "softening" of the damage.
It's funny looking back on that fear now. Password managers have become more mainstream, and we, collectively as users, have learned something interesting. One of the assumptions of that hacking fear was that it was more likely for a user's machine to become compromised than it was for the service's the user uses to be compromised.
Well, as it turns out, the opposite was true and the fear was unfounded. The vast majority of password leaks we've seen over the past decade have been due not to malware but rather server compromises.
Perhaps that makes sense in retrospect. But it wasn't obvious a decade ago. Back then viruses and malware were rampant on the internet. In many ways they still are, but not like it was back then. Those were the days of pandemics like the Blaster worm. So it made sense to be more afraid of your own machine being compromised.
But the tides have turned. The landscape of OS security has improved dramatically. The single points of failure offered by servers are far more valuable now than the unwashed masses of networked user computers.
Just a funny retrospective that I thought I'd share.
Something to consider is that malware-based compromises of personal systems don't raise as much brouhaha as corporate-level compromises.
So it's not often useful to take a password from an individual user and put it in a password list, but it is often useful to maintain persistence on their machine. Tools for doing that (RATs) are definitely common, both from random internet attackers and from e.g. angry exes with physical access to your device. But those aren't the attacks that e.g. HIBP is interested in, so if you're looking at data from people interested in password compromises, the effect is that you'll undercount client-side compromises.
You are underestimating how much malware there is out there, both on desktop and mobile. There are probably botnets that consist of >100k compromised machines.
So, I suspect the reverse is true with standalone PC's being more likely to be compromised, what makes this less noticeable is it's harder to automated extracting value from those hacked accounts beyond sending gmail spam etc.
PS: This is also why cryptocoin software on users machines is basically a non starter.
Now, malware is a criminal industry. People are making big money off of it. It doesn't make a lot of sense to target individuals when you could go after big, wealthy institutions. Same goes for government malware--they're looking to infiltrate and disrupt rival institutions and industries, not inconvenience private citizens.
Why do you rob banks?
Because that's where the money is.
That's absurd. I can count on one hand how many account I need to be extremely secured. Accounts that will actually affect me if they were hacked.
- Works - Bank - Paypal - Google - AWS
All have a different and secure passwords.
The remaining accounts that I care about but won't affect me much if they were hacked use variation of a single password.
The accounts where I don't care use a simple password.
If any one of the first one are hacked or leaked, well I was already screwed I guess, the service was hacked... just hope that they take care of it.
If I get hacked, I just hope that I catch it before I log into one of theses accounts, which funnily enough, doesn't happen that much (except for work, but that's mostly at works so not my responsibility).
If any other password get leaked? Well not too bad, I request a new password and that's it.
> The single points of failure offered by servers are far more valuable now than the unwashed masses of networked user computers.
It's true up until it's no longer the case. So many people use cloud backed password manager too... your point currently apply to them too.
You see the big leaks but you don't see the peoples that get their passwords stolen from their computers, it's not new worthy.
It's also the classical: it won't happen to me.
W3c and IETF or other similar clever folks really like security stuff and do lots of clever things to make us safer. So why couldn't we create a http browser/server authentication method that has something closer to a nonce-based challenge/response mechanism? If it were standardized, the browsers could even do some clever hashing of some peer addresses or other things that we think should be static. All of the browsers could still present a "username/password" field that looks remarkably similar to existing forms but never reveals the plaintext password to the peer.
As long as I could type my password into other browsers on other computers and still get the same result, it seems like it's moving things forward. It would be opt-in, traditional username/password forms would continue to live on for decades to follow.
So what am I missing? Presumably something like this was considered and ruled out?
EDIT: yes, of course, I forgot about the existing HTTP authentication mechanism(s) in RFC 2069. I don't know why it never caught on but the fact that the browser uses modal dialogues for these means a significantly different context/user experience.
In the end user experience vs security is a straight tradeoff.
Secondly, the digest authentication only supports MD5 anyway.
Really, the solution to exposing passwords to the endpoint is to do key-derivation client-side, with a server-provided salt.
Yes, the latter, please.
That sounds a lot like Secure Remote Password protocol: https://en.wikipedia.org/wiki/Secure_Remote_Password_protoco...
1. User provides username/password.
2. An exception occurs somewhere.
3. The stack trace from the exception is logged.
4. The stack trace includes the credentials.
5. The exception ends up in a ticketing system (Trac, JIRA, etc.)
6. Nobody notices for years.I'm not sure they fully understood what they were acknowledging, but I heard back later that they were impressed with my professionalism.
Having been in the identity space for a while and seen how various companies think about it, these kinds of problems will keep coming up...mostly at companies for whom identity is commodity. These companies will always give account security the minimum requisite attention. No one at Twitter gets excited about working on the login form.
I know OpenID was a bust and everyone hates Facebook Connect, but as an industry we need to figure out how platforms that view account security as a necessary evil can vendor that to people who take it seriously. Trying legal avenues to get people to take it seriously or finding alternative methods is what we’ve been trying for the last 15 years and it hasn’t worked.
For the average person (not worried about a megacorp or nation state attacking them), TouchID is also a possible "password replacement" for the primary authentication factor, although it comes with the disadvantage of not being rotatable.
It is exactly the same problem.
(switching sides of argument, I know)
I'm guessing you're referring to someone's ability to actually fix it -- in the case of logs, you can make a pretty simple regex to strip out all kinds of PII, and there really are a lot of arguments (e.g. proactively reducing cost of security audits -- if someone is reviewing your logs to figure out what happened, they might not want to see customer data).
> We are sharing this information to help people make an informed decision about their account security. We didn’t have to, but believe it’s the right thing to do.
Parag needs a Comms director to handle is Twitter account.
Hate to speculate, but it sounds possibly like perhaps a debug statement/log level had been enabled for testing and forgotten about?
Here's the tweet about it: https://twitter.com/TwitterSupport/status/992132808192634881
which links to that blog:
https://blog.twitter.com/official/en_us/topics/company/2018/...
> that they were exposed for “several months.”
I wonder what library was used, and which other companies use it which hasn't told their users. That'd let us know who isn't as transparent...
Hashing doesn't help much when you're logging everything first.
I'm tired of big shot Internet companies getting away with such bland disregard of basic security and privacy rules.
Brain surgery is hard. Mistakes happen. But after a few mistakes you probably should stop doing brain surgery altogether. At least the patients will get a higher chance to survive your surgeries by avoiding you.
In terms of security, handling passwords should be considered analogous to brain surgery. A single mistake undermines the whole thing. If you can't handle that, stop doing it, and let people do it who can handle it better.
If you banned every developer the first time they made a mistake their wouldn't be any developers.
If you keep and let bad developers in software security even if they made the same mistakes (and mistakes of others) repeatedly there won't be any security left. So what's the point?
We're not talking about a massive password breach, a bunch of script kiddies who found a database of plaintext credit cards by going to /admin.php and logging in with "admin / admin", or anything like that. We're talking about a mistake Github themselves made (and if you think Github doesn't know what they're doing in terms of security, I question your judgement).
Furthermore, when was the last time there was a major security breach at Twitter? You're claiming they're "keeping" bad developers and not learning from their mistakes as if this was a regular occurence for them.
And coming from me, I don't usually defend security breaches and malpractice. This doesn't really qualify. They made an official announcement, notified all users, even unaffected ones, both by email and on first login; that's more than you can ask them to do.
What bothers me about reactionary posts like yours is they give negative feedback to companies who actually do right by their breaches, which as is well known in the security field, is a matter of when, not if.
Imagine the twitter traffic and how much code they are dealing with. This kind of mistake can happen and will happen. Someone was surely held accountable which they do not need to disclose.
I've yet to see a company that didn't accidentally log passwords somewhere, some time.
It's 2018, no sane person would reuse their passwords across multiple sites which can and do get hacked.
All of my family does this :(. I think it's really common outside of tech savvy people. I've tried pushing them to use a password manager on their phone, or writing them down so they can use multiple passwords, but they'd rather just use a single one.
Years of bad advice didn't help either:
- Never write your password down
- Make sure you use at least one symbol!
- Make sure you use one capital letter!
Keeping your account secure When you set a password for your Twitter account, we use technology that masks it so no one at the company can see it. We recently identified a bug that stored passwords unmasked in an internal log. We have fixed the bug, and our investigation shows no indication of breach or misuse by anyone.
Out of an abundance of caution, we ask that you consider changing your password on all services where you’ve used this password. Learn more
Sigh. They appear to have confused hashing with asterisks.
> A bug caused the passwords to be written on an internal computer log before the hashing process was completed, the blog said.
So "related" in almost no way whatsoever, then? The state of technology reporting in the mainstream press really makes me despair sometimes.
The problem is that the mainstream press writes for mainstream users, and the state of science/technology understanding in the general public is the real problem.
> A bug caused passwords to be written on an internal computer log, the blog said.
> We mask passwords through a process called hashing using a function known as bcrypt, which replaces the actual password with a random set of numbers and letters that are stored in Twitter’s system.
Which is slightly better, I guess.
[0] https://blog.twitter.com/official/en_us/topics/company/2018/...
No they didn't. Or else the excerpt you quoted would have said literally that, i.e.
> The glitch was related to Twitter’s use of a technology known as “hashing” that masks passwords as a user enters them by replacing them with asterisks.
While the description of "hashing" is tortured, it seems to be technically correct, if you don't make specific assumptions about the phrase "that masks passwords as a user enters them by replacing them". But yes, the normal interpretation of the present tense would be that the characters are being hashed as the user types in the password, rather than after the form submission.
Why isn't the password hashed and salted client-side, with the salt being sent from server to client?
I understand that hashing in the client side would only substitute the user password for a new one. This alone sounds like a win to me, as it at least contains the damage somewhat in case of leakage, specially if salted.
EDIT:Sorry, I just found the exact same discussion in this thread.
I have trouble believing no passwords have been misused by insiders - sure, there wasn't any large-scale misuse but I am sure someone poaching a few passwords here and there for later mischief (once everything has settled down) would have gone unnoticed.
Yahoo’s case was worse though, since it was hacked.
Based on that I'd say it would be pretty safe to disclose a breach and reset all passwords; if your service is relevant your users will stay with you, and if not then not disclosing a breach will only buy you time before the inevitable happens anyway.
Twitter did disclose this, through email and on first login. Anyone they'd lose because of the breach is long gone and I also think it's probably next to nobody.
- server publishes a public key on a fixed url
- your JS framework (react/angular/form handling/whatever) is modified to, whenever you query the value of a input[type=password], to return a value encrypted with the private key. (as close to the reading of the field as possible, except for eg. 'repeat password' handling)
- your serverside code decrypt the password as close to their 'VerifyPassword' and 'UpdatePassord' implementation as possible. and fail if they receive an unencrypted password.
This would make it very hard for intermediate steps (middleware, RPC/REST/JSON decoding) to accidentally spill the password, even if they fully log the traffic, and is something frameworks should be able to help you enforce.
Seems like a pretty simple system and absolutely nothing that has to be secured is stored server side. Is there some clever reason I'm missing that this system fails?
----
Even better you can also salt the password->key schema based on something like the username, making table based attacks infeasible.
At the very least they should have a security alert at the top of your feed or something.
Edit: Looks like they are alerting users.
To add to the speculation in the linked thread, I _did_ recently reset my password, but it was on 28th February.
Now I'm wondering if I need to change it again... does anyone know what time the bug was patched?
This is a bad idea, but if you are required that your uses have passwords, you could use their password to seed an elliptic curve private key to create the user key on the fly.
When you first set your password, it sends the public key to the server, which is then stored. If that key was intercepted, it's not a problem.
Then the server challenges by sending a random code, which the client uses the private key to encode, sends back, the server then decodes with the public key. That guarantees that the private key is known by the client, and thus the password is known, but the private key never leaves the machine.
The second time you go to the site, you get the challenge, and respond. Neither private nor public key is transferred.
Do I understand that right?
data leaks of course are always a breach and must be reported. that doesnt mean you get a fine though.
The most likely downside is that without JS it won't work.
That's not to say it's a bad idea, though. I have used client-side hashing in the past to allow passwords of arbitrary length while using finite network resources.
Also you can't get away from sending the shared secret during registration. Even with asymmetric cryptography you have to rely on the TLS to make sure you exchange the real untampered public keys.
In Yahoo's case, that might have forced Marissa to actually keep a cybersecurity team and not cut them when she knew the systems were in danger of being compromised. We aren't getting any jail time, but hefty fines that don't stifle growth, just punish negligence and carelessness that are codified and don't need long court hearings to pass are a must.
Security is expensive, breaches have to be made more expensive. That's the difference between a company's management thinking about security as a checkbox they have to fill as opposed to an ongoing investment meant to reduce risk.
Section 508 of the Rehabilitation Act legislated that the government purchase accessible software.
HIPAA legislated that your medical data be kept secure.
Minnesota, Nevada, and Washington have enshrined some or all of PCI DSS into law: https://en.wikipedia.org/wiki/Payment_Card_Industry_Data_Sec...
A little farther afield, seat belt technology has been legally mandated to be included in most automobiles sold in the United States since 1968: https://en.wikipedia.org/wiki/Seat_belt_laws_in_the_United_S...
No matter how advance our technology is or the security measures we take, any system connected to internet somehow will have a leak or an intrusion or something that compromise security.
For example, a common refrain on HN is how centralized the internet is becoming. Do punitive damages for mistakes prevent mistakes? How likely is it to create an environment where the only organizations that exist are those that can afford mistakes? Is that worth it, especially in the context of some Twitter passwords that were logged internally?
Also, one of the most important awakenings that need to happen in light of recent events is the personal responsibility of who you share your information with. It's just as important as the question of what an organization does with your information.
If there were no trade-offs, then we could fix everything with legislation.
For example, I can imagine kneejerk legislation that would simply ensure that next time this happens, Twitter just keeps their lips zipped. Not exactly an improvement.
Can you pitch an example of what this legislation would actually look like? And how it would differentiate between something like the Experian leak and some (oh no) Twitter passwords getting logged internally.
Can't leak a password if the only thing you have to begin with is a hash...
Is that the state of infrastructure engineering? The cynic in me wonders if it's related to the trend to take developers with some systems knowledge and turn them into Ops/SA's.
// logger.log(request.payload);
Anyway, I guess you're right, I simplified. It is kind of a valid point, although if I was this paranoid then I would never leave the house. I just wanted to say that if the web wouldn't have been built on a giant piece of s*, for example password authentication, none of this would have been necessary.
... maybe in a vacuum. It's possible that if you had all the facts, you just might agree with their course of action.
It just sounds so incompetent.
debug logs is that necessary evil you need to troubleshoot pesky bugs. Unfortunately some of these debug tools need to be turned on in a live environment to capture those logs for debugging. But also Unfortunately, we are humans and we concentrate on fixing the bug and forget to turn off logging or log unnecessary data.
One possibility is an HTTP server on the request path after TLS termination. But then why is an HTTP server logging the request body?
My guess would be some sort of instrumentation process was blindly reading data in memory without distinguishing what the data was, but produced logs that incidentally included passwords.
POST request comes in from the client. Full URL and request body is logged. Sometimes for simply troubleshooting, sometimes for security reasons (e.g., wanting to know all data coming in so that it's possible to identify security holes after they've been exploited).
POST request comes in from client. Frontend server makes a GET request to a backend server, and the password ends up in the standard request logs. In one case, I've seen this happen because the developer thought path variables were cool, so every API they wrote looked like /a/b/c/d/e. Sigh.
Over-zealous developers who think it's appropriate to log all function calls with parameters for trace level logs, or a framework with the same opinion automatically applies such tracing and logging over the whole code-base.
No-one notices because no-one uses trace level logging, until one day another developer is tearing their hair out because they can't reproduce a bug that is only occuring on live. It's an urgent bug that needs resolution asap. So this developer turns on the trace level logging and eventually finds and resolves their bug.
Being the careful person they are, they turn off the logging and go away happy.
Meanwhile they've unknowingly produced a few gigabytes of log outputs which happen to include plaintext passwords.
That's just one of many different scenarios where people acting in 'good faith' can still lead to bad outcomes. That is why a "PUNISH THEM!" attitude to this kind of incident is not helpful.
But without persistence on the client side you wouldn’t be able to do salting in the first hash (where do you store the salt?)
At most, it would seem to prevent weak passwords from being passed directly to bcrypt, but salting should solve that in a similar way anyway, and anyone brute-forcing a copy of the database can incorporate the same weak hashing logic
That would effectively create a one time “password” for transmission from browser to database. In a case like this one, where sensitive text transmitted from the client leaked into logs, it would be a non-issue. The sensitive string in the logs is a temporary hash that would be useless shortly after discovery, since it was derived from an expired nonce.
It effectively becomes a real time scrubbing system with 100% coverage, because the passwords are “scrubbed” by design, and do not depend on explicit detection code in some scrubbing mechanism.
From the authorization side, there is a threat, because if your table storing hashes is compromised, attackers just have to supply the stored hash to the auth endpoint and they get to login as anyone.
A combination of hashing on the client side (or immediately once the pw hits the endpoint) with something cheaper followed by a more intense bcrypt/scrypt afterwards might help a bit with the tradeoffs.
You'll notice you've been downvoted to hell and the comments in reply to yours are apologists and excuses. Not a coincidence.
FizzBuzz
This isn't a bug, it's incompetence. Remember folks, never, ever use the same password for two different websites. Assume all websites store your password in plain text even if they hash it at some point. With logging and tracking on steroids these days, I expect "bugs" like this to be commonplace.