1. The credentials were resent to an already compromised email account
2. This is an isolated case
3. Established procedure was not followed
4. With thissaid, we've used this as a learning example and additional training has been provided to the individual involved
5. Anyone with any self-managed server with ANY provider should always keep their own multiple backups
---
Disclaimer: I'm [not] CIO @ Namecheap
We messed up, big time. While we handle 1000s of live chat sessions everyday without issue, I realize that even one breakdown in security protocol can cause huge problems and a loss of trust for our customers.
In response to this isolated case (in which our established procedure was not followed), we will be creating additional training material for all our live support staff. Additionally, we will be exploring technical solutions to try to make this kind of breakdown much harder. Mistakes happen, but if we can prevent them, it is worth doing.
We also would like to take this opportunity to remind folks that any self-managed server (regardless of provider) should always be backed up in multiple places. For information on how to do this with Namecheap, we've published a guide here: <link>
I've reached out to author of the post already by email and we are working to help them resolve any outstanding issues.
Overblown/fake apology is not very informative - it's hard to say what exactly can you trust in it.
* Actually apologize in a human way
* Show empathy by identifying the impact of what happened to customers (not your impact internally)
* State action items that you've created, even if they are just in 'evaluation' state
* Indicate that the specific incident in question is being handled outside of this forum
* Take responsibility for things even if you shouldn't "have to"
http://blog.statuspage.io/why-public-apologies-suck
Some important bits:
>4 PARTS OF A BAD APOLOGY
- Justifying the offending actions or words.
- Blaming the victim.
- Making excuses.
- Minimizing the consequences.
>8 PARTS OF AN EFFECTIVE APOLOGY
- You actually have to use the words I’m sorry.
- Acknowledge that you messed up. (As in, “I take full responsibility for my words.”)
- Tell the person how you’ll fix the situation.
- Describe what happened, but without foisting the blame off on someone else.
- Promise to behave better next time.
- Make sure the person knows you know exactly how you hurt or inconvenienced them.
- Much like the first rule, it’s important to use some version of the phrase “I was wrong.”
- Ask for forgiveness.
This is not the correct solution. What's to prevent the next new person from making the same mistake?
If it shouldn't happen, don't make it possible to happen. Put in place a technical solution that doesn't allow it happen. And if there is some special case where it still needs to be possible, make it that it needs a secondary signoff from a senior team member.
People will always be fallible.
And how many people at Namecheap do you think aren't aware of this by now?
But yes, technical solutions should go in but those take longer to implement. Among other things, it seems to me that re-prompting for the account password might be a good idea before any VPS reinstall/reinitialization that's going to wipe an existing VPS (not that it would've helped much here).
You also didn't mention all the terrible things the OP pointed out that someone can do with just your password even when 2fa is enabled.
However, it's relevant to the story because there's a huge difference between sending a password reset to the email already listed on an account vs. resetting it for any random person who starts a chat.
This doesn't excuse their other issues, but it makes the customer support rep's behavior a bit less awful, even if they still violated protocol.
Wouldn't it make sense that support staff can only generate and send out password reset mails if the PIN/password has been entered into a form? I don't know the term for this - like "coded procedure".
In this case, the support staff wouldn't even needed to be trusted in the first case.
Why have a procedure if your support doesn't follow it? Even if you have a procedure, everything falls apart when it isn't followed. This is the same as having no procedure at all.
Note: I have no direct or indirect relationship with Namecheap at all.
EDIT: My use of the term is a bit strong. I feel frustrated that company execs cannot explicitly admit a mistake or apologize. I should have worded it differently.
EDIT2: just for Tamar. By explicit I mean literally using the words "sorry", "apologize", or "mistake". What we have is the standard corporate nonapology.
EDIT3: congrats to Tamar for being promoted to a Namecheap executive!
Usually there are backup options included in the plan for upsell possibilities with these kinds of providers. Really, you should not expect a service that has 'cheap' in the name to offer any kind of backup.
An example would be if some service stored credit card information temporarily while waiting for transactions etc. to process but then purged each record after two weeks later. A compromise of the backups containing, say, weekly snapshots could then contain 90% of a client's ever-stored financial information whereas a compromise of the main site might only reveal a couple percent of them.
In reality though, more companies do this than should be allowed. I worked for an ISP in a previous life and even on the super cheap shared hosting there were companies that were making a decent turnover and then using the cheapest possible hosting for their email/site. The quantity of these companies was a significant number too.
Especially when they were kicking off on the phone due to inevitable maintenance/downtime. Trying to appease customers that turn over 10 million a year and pay £5 a month for hosting is a bit wtf. You pay for what you get... that's no different in hosting.
They never spell this out for you though, usually they imply that their service is just as good as their pricier rivals. As a result, many people get burnt before they get savvy. Some never get savvy, they just get turned off to the industry.
Not sure what the alternative is. I suspect a company that did clearly spell out their pros and cons would risk having stunted growth or go out of business entirely.
"If it's too good to be true, it probably is"
and
"Cheap, Good, Fast - pick two"
The opposite of what I'm suggesting is that people - individuals/companies - do not look after their own backups. That's a dangerous precedent.
This is what they read as the company's response to this loss:
"Anyone with any self-managed server with ANY provider should always keep their own multiple backups. Dumbass."
Note the change I made at the end to reflect how some people [who are empathizing with someone who was attacked and lost their property] will interpret that statement. Did any of that statement help the situation at all? Did it help customers feel better? Or did it have the opposite effect? Would this be considered a good way to engender goodwill for your brand?
Now consider this reinterpretation of the statement:
"With self-managed servers, it is good best practice to keep multiple backups for yourself, no matter who your service provider is."
Unless you sign up for a managed service that claims to include backups or whatever, you are responsible for your own backups. What's controversial about that?
I mean, if you go for a stroll through the roughest neighborhood in town, unarmed, by yourself, at night, and you get mugged, is it wrong to point out that going for that walk was stupid? Saying so doesn't mean the the mugger isn't guilty or that what happened is right in any sense. It's just acknowledging reality.
In other worlds however, the problem is usually too widespread. You might get a lot of attention, comiseration, etc. but in the end, being reckless goes against survival. People who point this out should not be shushed for pointing out what you need to do to survive.
Its amazing to see that this "victim blaming" mentality is growing in Brazil. Violence here is out of control. You might get mugged/shot/kidnapped for no reason, or not displaying any wealth. Having been kidnapped myself, and chatted with the kidnappers, they do look for signs of wealth before pouncing. Therefore, yes, the victim does has an ounce of control over their risk and it's not wrong to point that out.
It does not solve violence, and attackers will just look for other victims regardless of their reward estimate. However, would you tell your children not to not show affluence/vulnerability in shady places just because you don't want to "victim blame"?
Yes, this is the textbook example of victim blaming. Placing any amount of blame on the person who is the victim in this situation is saying that they don't have the right to walk down a street and not be mugged. I am admittedly not the best at describing this because up until recently I had the same thought process as you. I would encourage you to find better explanations than what I can offer and be willing to have your beliefs challenged.
Then "victim blaming" is a meaningless concept and we should quit using it. Because if I choose to do something stupid, I do bear some responsibility for the outcome, even if somebody else violates my rights. That doesn't absolve the other party of course, which is my point. That is, you can blame the perpetrator of a crime while also pointing out that the victim could (possibly should) have done things differently.
Placing any amount of blame on the person who is the victim in this situation is saying that they don't have the right to walk down a street and not be mugged.
It isn't "blame" for the actions of the other person. Why wouldn't you point out the stupidity of knowingly putting yourself in a dangerous situation?
You can say a victim is stupid without giving any vindication to the person in the wrong.
Namecheap are saying "be responsible for your backups, but yeah we screwed up on our security policy" - They are seperate things, that the victim here has conflated, but are seperate problems (in regards to namecheaps offering).
Also, it's stated in the knowledgebase that it is advisable to set up server backups of your own if you do not have a managed server: https://www.namecheap.com/support/knowledgebase/article.aspx...
Although backups are #1 item on any list of best practices, making an easy, and tested, implementation method would be a good practice on your part.
Even my managed services have offsite backups. Better be safe than sorry, I always say.
I don't think the best way to respond to a public vent is "Here's what you should have done instead". Responses might be technically correct but they lack empathy for the customer.
For example, let me give you a look at what my Windows hard drive looks like.
My important files are stored locally, on Dropbox, and on CrashPlan. Some is also on Google Drive. I also run an offsite backup of my own to another local Linux box.
Don't make this specific to Namecheap, @kelukelugames. It's always smart to have good recovery systems in place. If you care that much about your data, you will protect it at whatever cost.
So yeah, I repeat, better to be safe than sorry. Your mileage may vary.
To be fair, your third edit was only for me. And that was the only comment I was replying to.
As I said, we are working to respond to hundreds of comments across dozens of platforms. I certainly respect your distaste in the more rushed responses in order to address all of the deluge, and that is why we were also simultaneously working on a longer and more thoughtful response that speaks for all of us at the company via that blog post (in a far more emotional tone).
It certainly is difficult to envision the challenges of responding to dozens of responses if you're not in our shoes. But I genuinely thank you for the feedback - and we're noting this (as well as the feedback all have been sent to date; a lot of that was factored into policy adjustment and our blog response) for handling it differently next time.
And although it's not you area, can I just say that Namecheap's website is just way too slow since the redesign? I appreciate that you even did a redesign, but for some reason it's one of the slowest websites around. I don't know if it's because of the large images you use on your pages or what's the problem, but I suggest you fix it. It may be losing you customers. A web services company's site should be snappy.
It sounds like you are confirming that this incident did happen and it was your fault for not following your procedures.
I am not a lawyer, but since there was signification loss, it would probably be in your best interest to offer better reparations.
OpenDomain has several domains that are on NameCheap - I will transfer them immediately since it appears you do not care about customers.
We have apologized, admitted mistakes, and made tremendous internal change to move on for the better. We would not do that or even post here if we didn't care.