ProtonMail pays $6k ransom, gets taken out by DDoS anyway
arstechnica.com
arstechnica.com
Please. Even if your business will suffer it will suffer a lot more if you do pay since now it is known you'll cave. Also: you are making the problem larger for others.
At around 2PM, the attackers began directly attacking the infrastructure of our upstream providers and the datacenter itself. The coordinated assault on our ISP exceeded 100Gbps and attacked not only the datacenter, but also routers in Zurich, Frankfurt, and other locations where our ISP has nodes. This coordinated assault on key infrastructure eventually managed to bring down both the datacenter and the ISP, which impacted hundreds of other companies, not just ProtonMail.
At this point, we were placed under a lot of pressure by third parties to just pay the ransom, which we grudgingly agreed to do at 3:30PM Geneva time to the bitcoin address 1FxHcZzW3z9NRSUnQ9Pcp58ddYaSuN1T2y. This was a collective decision taken by all impacted companies, and while we disagree with it, we nevertheless respected it taking into the consideration the hundreds of thousands of Swiss Francs in damages suffered by other companies caught up in the attack against us. We hoped that by paying, we could spare the other companies impacted by the attack against us, but the attacks continued nevertheless. This was clearly a wrong decision so let us be clear to all future attackers – ProtonMail will NEVER pay another ransom.
That would give their customers time to batten the hatches and/or migrate off the system for the time being while sending a clear signal that they would not pay anyway.
This is a tough situation to be in but putting your customers in control of the company (and in a democratic way no less) is not the solution. What about those customers that decided (rightly imo) against paying?
Companies such as these should have an up-front item in their terms of service indicating that they would never pay a ransom, that way they would be clear to both their customers and their potential attackers.
I think they were put under pressure by other companies using the same IPS, not their customers.
Great in theory, but surely nobody "elsewhere" will host you securely if hosting you means all their other customers get hosed.
"the attack against ProtonMail can be divided into two stages. The first stage is the volumetric attack which was targeting just our IP addresses. The second stage is the more complex attack which targeted weak points in the infrastructure of our ISPs. This second phase has not been observed in any other recent attacks on Swiss companies and was technically much more sophisticated. This means that ProtonMail is likely under attack by two separate groups, with the second attackers exhibiting capabilities more commonly possessed by state-sponsored actors. It also shows that the second attackers were not afraid of causing massive collateral damage in order to get at us."
Protonmail could just be talking this up, but if your ISP's (or AWS's) fancy countermeasures don't deal with this, why would they keep you? And why would any other ISP want or accept your business?
There are very good clean pipe services available; the major limitation is that the clean pipe provider must have enough capacity to absorb any attack... something that can be quite difficult unless you are someone like L3.
However, the good clean pipe services are all very expensive. (I don't mean the "http only" service like cloudflare; that is a very different sort of thing.) - this is because of that aforementioned limitation; you need a lot of headroom in your bandwidth to run a clean pipe service.
But yeah, amazon charges a lot more for bandwidth than you'd expect to pay direct from a transit provider at small-ISP scale, so I would hope that they have enough capacity and technology to filter fairly large attacks.
Yes. And what does a provider do when a customer is getting hit so hard by a ddos that it is pushing their other customers offline? they blackhole the target at their upstream (usually starting on a per-IP basis, but that will widen as the attacker shifts the target)
So... most likely, the isp said "if this continues, we will need to finish the job and shut you off" - which is what every other ISP is going to do in the case of an attack that is large enough to knock the ISP in question offline.
Check out the legalese on your hosting contract; everyone reserves the right to dump you as a customer in these sorts of cases.
I'm interested in whether the ISPs have any form of protection against the disruption caused by a customer of a shared service coming under a criminal attack of this kind.
Actually, it makes no sense to not follow through because that is their business model.
So you just simply do not pay extortion fees unless you want to become part of the problem.
In the case of an encrypted filesystem that means you will have to restore from a back-up (which I assume (naively maybe) that you have). And you chalk the whole thing up to your education fund. Paying up is simply wrong.
Right, which is why "never pay extortion fees" doesn't make much more sense for combatting this stuff than "never click on spam links" makes for combating spam. It's unrealistic to think we will convince enough businesses to altruistically not pay extortionists, just like it's unrealistic to think you'll get your grandmother to stop clicking on spam links. You need another solution.
But once you as a business pay an extortionist you have just taken on another partner in your business, who will do none of the work and who will take almost all of your profits. So paying out of pragmatism will actually have the exact opposite effect of what you intend to achieve (to make the problem go away).
A good parasite does not kill the host, merely takes all the resources they can get and it certainly won't stop with one attempt at extortion. And judging from the blog post linked they learned their lesson.
edited for clarity, thanks ghotifish.
>> The only way spam will go away is if everybody will finally stop responding to spam.
> Right, which is why "never pay extortion fees" doesn't make much more sense for combatting this stuff ... It's unrealistic to think we will convince enough businesses to altruistically not pay extortionists,
jessriedel is not saying it's altruistic to pay, it's altruistic not to pay.
http://www.amazon.com/gp/product/1492603236 http://krebsonsecurity.com
Anything less than that is not a backup but a mirror and mirrors while useful are not at the same level of security that a backup is.
Some copies are backups, but not all of them and most copies on spinning or re-writeable media especially when they are networked are not actually backups. Somebody tell backblaze ;).
That's true but for the individual payee it can make sense. Trying to get the ransomers back can work. They'll keep at it till they figure they can get harmed.
There wouldn't be any point in paying a $1m ransom for a $50m boat if it was going to get caught by 49 other pirate groups on its way to safety. So when a ransom has been paid, pirates escort a boat to safety.
DDOSers can't offer any such guarantee; it's not like one DDOSer can stop another going after the same victim. And to my knowledge there aren't any websites where victims can post reviews saying whether the same DDOSer targeted them again later on.
There is a chance they could be 'honorable' thieves and desist, but it's likely having had someone cave in once, they'd cave in again, and again... So, it only makes sense as a delaying tactic, in the long run it's mostly a losing proposition, unless you're setting them up for a sting or something.
I've been in that position (twice) and in both cases was able to reverse tables on my opponent. It could be that I'm lucky but I think that these operations only work because there are a lot of people that cave in when they see a letter on a lawyers letterhead regardless of the merit.
-Winston Churchill
This goes for any 'in the cloud' data that you might have. In the end it's your data and your company that is at stake. Not all data wipe-outs are malicious, sometimes accidents do happen.
http://www.theregister.co.uk/2014/06/18/code_spaces_destroye...
http://www.dwheeler.com/essays/scm-security.html
Orange Book systems' (1980's to early 90's) used air gaps and/or paper backups in safes. OpenCM, a robust SCM by Shapiro et al, mentions that among other things:
https://web.archive.org/web/20060315100242/http://opencm.org...
SoD alone would go a very long way to close some of the larger holes (dd question: who has access to your backups?) but even that is something that a lot more people seem to be aware of than is put into practice.
The amount of trust placed in the hands of a very few people is scary, and to do all that without real backups is something that would keep me awake at night if my bread and butter depended on it.
you would be shocked at the number of people who get upset when you advise them to make their own backups, and interpret this as an indictment of the reliability of your own backup procedures.
e.g. "isn't that what we pay you for???"
nevertheless, do it anyway and let them fume. there are no prerequisites for running a business and you'll find that many absolute morons are at the helm of some nominally successful businesses.
A company of any size can continue on even if severely crippled with nobody left who understands how anything works. I've seen it time and again - also even where I've felt I was important.
Minimal viable product and vendor lock ins are powerful real world things.
'So basically ProtonMail said "We're incompetent and fund criminals… give us money."'
"ProtonMail should apply for a refund. Or at least store credit."
Would it be in the legitimate interest of the public as a whole for a third party (possibly governmental) to carry through on the threat as soon as the ransom is paid? This would be to the detriment of the victim, but reduce the likelihood that future ransoms would be paid, and thus eventually might reduce the number of future victims.
Might that be what's happened here?
Huh, interesting. If third parties have attackers' bitcoin address, they can also pay the ransom themselves.
This way:
(1) companies that pay ransoms are AWLAYS punished and it never causes an attack to stop (2) no attacker ever gets paid a ransom
(The email deliverability problem doesn't help matters, of course.)
If you're just talking about availability, then yes, but this was a sustained DDoS that took their servers down for hours. While the email protocol does insist that the sender should queue and try later, having no new email for hours is not really what people want out of email.
"Cost estimates for these solutions are around $100,000 per year since there are few service providers able to fight off an attack of this size and sophistication. These solutions are expensive and take time to implement, but they will be necessary because it is clear that online privacy has powerful opponents."
No shit lol... Not a good sign that they're already in reactive mode. On other end, that MyKolab hasn't gone down might mean they're already compromised or just not targeted by this attack. I wonder what it is. They're just a GPG carrier in a semi-neutral jurisdiction in my usage, though. ProtonMail would've been, too, but I figured they'd be more likely to have service issues.
Here's a specific example where I try to make a step-by-step guide for high assurance Tor without knowing its internals. Just drew on my prior work:
https://www.schneier.com/blog/archives/2014/09/identifying_d...
Hope what High Assurance Security takes is more clear now. Unless you get lucky (eg GPG), you need high assurance to resist TLA's successfully and that might just be delaying inevitable. Still need monitoring & tamper-detection.
"Somebody with great power, who wants ProtonMail dead, jumped in after our initial attack!" "We have no such power to crash data center and no reason to attack ProtonMail any more!" "WE DO NOT HAVE THAT POWER! NOT EVEN CLOSE!" "We are not attacking ProtonMail! Our attack was small, directed at their IP only and lasted 15 minutes only!"
I don't believe Protonmail have said they have received any more requests for money, so that would go along with the above. I agree that it was silly to pay the blackmailers, but there is some reason to believe that these are two separate attacks.
Props to cloudflare for standing by to help out in that particular instance, absolutely fantastic.
I'm not sure what to think, but I can easily understand why they did pay. It's easy for others to say what would be best for the industry, but when you are the one suffering and your ISP is angry at you, and you can pay a small sum to (possibly) make the problem go away, your opinion will change.
There are two kinds of protection, basic HTTP/HTTPS and DNS only (done with DNS and CDN like servers co-located at peering points), and traffic filtering that is done through BGP with and a GRE tunnel. While you can get basic HTTP/HTTPS and DNS from CloudFlare for $200/month on a business account, what ProtonMail needed was a BGP/GRE which at it's lowest price is a multiple and an order of magnitude more expensive.
Not only that, there is a power imbalance that shouldn't be ignored: the criminal has more experience in these kinds of confrontations than you do. Sam Harris has a very good article on this topic[1]; while he is discussing violent interactions on a personal level (e.g. mugging), the principles apply to many situations. The short version is that the criminal is trying to draw you onto their turf and to play by their rules. Almost always you will only make your situation worse when you let the criminal set the rules.
[1] http://www.samharris.org/blog/item/the-truth-about-violence
I am amazed about how many people are making this claim confidently in this thread. It's clearly wrong. Very, very often it's definitely worth the cost, because very often you will never see the same criminal again. Consider:
"Don't pay ransoms, because (1) you'll get extorted again once the criminal knows you're an easy mark and (2) if everyone always refuses to pay, criminals will have no incentives to try and extort."
versus
"Don't pay muggers, because (1) you'll get mugged again once the mugger knows you're an easy mark and (2) if everyone always refuses to pay muggers, muggers will have no incentive to mug."
Yes there are cases, like if you're the government, where you are very long-lived and your reputation is reliable such that having a stated, followed policy of not being extorted works. But for individuals, it's just not feasible most of the time. You probably won't see that mugger/extorter ever again, and it's very unlikely that most victims will refuse.
But extortion is different than mugging. See, in extortion you have a perceived weakness other than that you fear for your life and that weakness has subscription possibilities, unlike mugging people. For instance one simple defence against muggers would be to have nothing on your person. Hard to mug you in that case. But since the ransom victim can't really change the nature of his business (short of removing themselves from being online) they will always be open to a replay.
Individuals are not the parties being extorted here, it's companies with some degree of success and visibility. I pretty much guarantee you that every larger entity online has either been prodded by extortionists or will be prodded in the near future. This is a very large business and everybody that pays makes it a bigger issue because of the perceived easy money drawing in ever more prospective extortionists.
Muggers != extortionists. Blackmailers are extortionists and they always come back until they get stopped through some other means (for instance the authorities) or until you tell them to do their worst.
In the case of one Dutch bank this led to intermittent outages over the course of several weeks but eventually they got things under control and there hasn't been a problem since. If on the other hand they had paid I'm pretty sure that they'd be paying a nice monthly protection fee. "It'd be a terrible thing if something happened to that nice website of yours.", it's just the same tactic as the mob employs against shops.
Additionally: How many of the people who do not pay these ransoms do you really think are hit again?
Huh? People report extortion and muggings to the authorities routinely. Combining that with surveys to estimate non-reports should allow us to get a very good estimate.
> How many of the people who do not pay these ransoms do you really think are hit again?
About the same number as people who do pay: Few.
https://news.ycombinator.com/item?id=10482242
I think this is a good example of why this is bad advice.
With a DDoS, there are almost no advantages to paying the ransom. Much better to spend the money on DDoS mitigation instead, to help now and in the future.
Also, the FBI wasn't making an official statement. It was just an off-hand remark from an agent, recommending technically ignorant people who desperately want their files back to pay the ransom.
(damn, it does sound like a cool domain)
From the fact that it knocked off their upstream providers also means it was probably just a simple volumetric attack like an NTP or DNS reflection attack. These are relatively easy to defend against.
I work for an ISP that gets hit with 5 or 6 of these a week, but because of the mitigation strategies we have in place our customers don't even notice...
I don't really understand the logic behind setting up with a Swiss datacenter with zero (or very little) DDoS protection. It is pretty much guaranteed that China will DDoS you if you are in any way involved in helping dissident groups.
We don't proxy smtp. There are solutions to deal with that in a hybrid way, though.
Reminds me of when Uber had that surge pricing scandal during the Sydney hostage crisis.
Unfortunately the only safe play is to give away the service for free (for duration of the attack). Which could be a solid marketing strategy, cloudflare's price point is reasonable enough that many would stick with their service even after the attack was over.
For all the people getting nasty and arm chair quarter backing this on little to no information or trying to claim credit for things they did not do- understand that once you start working in venture funded startups pretty much everyone knows each other and many people have worked together before.
Additionally, I don't see ProtonMail as the kind of company that'll let other third parties terminate their SSL connections/proxy all their traffic.
In fact is 100% of people never paid a ransom the attacks would not be funded
Publicly speaking about paying ransoms is very unusual.
ClouldFlare are a bunch of great guys. And, they wouldn't do any of that unless they were delivered a National Security Letter forcing them to.
If ProtonMail signed up with CloudFlare, like HushMail did, ProtonMail would have no way to know if these types of code modification attacks or metadata collections were happening.
And, as people saw with Hushmail, since CloudFlare does not do SMTP proxying (filtering/challenging) a DDoS could have still taken down ProtonMail's mail servers offline. While CloudFlare allowed Hushmail to get it's website back online, mail to my Hushmail account is currently delayed by several hours due to DDoS of their mail servers.
From https://hushmailstatus.com/ :
"We're investigating reports of incoming and outgoing email delivery delays. We'll update this page as more information becomes available."
And that is called asking for Dane-geld, And the people who ask it explain That you've only to pay 'em the Dane-geld And then you'll get rid of the Dane!
It is always a temptation for a rich and lazy nation, To puff and look important and to say: -- "Though we know we should defeat you, we have not the time to meet you. We will therefore pay you cash to go away."
And that is called paying the Dane-geld; But we've proved it again and again, That if once you have paid him the Dane-geld You never get rid of the Dane.
It is wrong to put temptation in the path of any nation, For fear they should succumb and go astray; So when you are requested to pay up or be molested, You will find it better policy to say: --
"We never pay any-one Dane-geld, No matter how trifling the cost; For the end of that game is oppression and shame, And the nation that pays it is lost!"
- Rudyard Kipling
Computer security is of course a whole different thing.
Seemed like your odds of dying were pretty high either way. Give him all your stuff and starve. Or say no and get beheaded..
I'm not sure about the starving part, would you have any references? If a city surrendered he did not sack it. If it did not - yeah, bad.
I must come to Britain's defence here - it's behaviour was normal in those times but it did eventually give up most of its "ownership" without actually being defeated in wars. That was pretty amazing.
By modern standards, British behaviour was despicable, but a lot of the invaded countries got enormous benefits - rule of law, economic infrastructure, transport networks etc. Being invaded (not just plundered) by major cultures has generally had good benefits - in the long term - for the invaded nation as they get a lot of the characteristics of the stronger nation.
Again, keep in mind that this is not the modern way of looking at things, which is why we have the United Nations and other international organizations.
As for Britain giving up its ownership: To this date formally England lays claim to a whole bunch of places that they have colonized and in some cases it has gone to war to keep that situation as it is.
That the UK gave up India is a pretty complex affair but you can bet that the 'let's fight' option was only taken off the table when someone did some basic math.
Whether or not the invaded countries got 'enormous benefits' is immaterial, we do not live in the alternate universe where India was not a British colony, in which universe India may have been better off or it may have been worse, we simply can not know.
All we do know is that in this universe we (nowadays) take a dim view of such colonization, including those colonizations in our collective past. That some countries were 'not as bad' as others and that they left the places they invaded (and usually plundered) in some ways in better shape is imo immaterial to that.
> As for Britain giving up its ownership: To this date formally England lays claim to a whole bunch of places that they have colonized and in some cases it has gone to war to keep that situation as it is.
Examples in post Suez history include?
"Other places" do not exist unless you name them. Even the Chagos Islands, the UK's most questionable overseas possession, have only been fought over in the courts.
Any actual places?
Yes, all of Europe was engaging in the monster known as colonialism.
That doesn't make it any less morally repugnant.
> it did eventually give up most of its "ownership" without actually being defeated in wars
It gave up its ownership only after being devastated by two World Wars.
> By modern standards, British behaviour was despicable, but a lot of the invaded countries got enormous benefits - rule of law, economic infrastructure, transport networks etc.
None of these things couldn't have been achieved without the British. And all these things came with a cost.
This justification has been used for colonialism time and time again. We saw this last year in Ukraine. It doesn't make it right.
e.g. Colonialism in India was an incredible amount of good luck and some wily statesmanship, and not due to lack of technological progress. Once the country was colonized, the British had an ulterior interest in preventing industrial development and the concomitant economic progress.
Those "benefits" stopped innovation and progress of their own cultures. And it led to the present day of conflict between traditionalists and progressives, slowing harmonious progress indefinitely. The traditionalists would have eventually progressed to a more "civil" society in a different way. And then we would have a far richer diversity than today's system.
http://www.amazon.com/French-Intifada-Between-France-Arabs/d...
As an example, Japan nor China was never colonized, they were totally ass-backwardian to late 19th century and appear at least to foreign eyes quite modern nowadays.
And as for Britain "giving up" their claims, they simply couldn't afford to keep India after the Indian military rebelled, and without India, they simply had not enough colonies to make profitable quickly after the devastation of WWII.
It was about the Philippine-American war. 2 days after publication in America, it was read in the Senate to argue for the US to end the war.
One of his more famous stories, The Man Who Would Be King is about two white men who manage to convince an Afghani tribe they're gods. It becomes undone, when one tries to marry one of the women, she attacks him drawing blood, and the tribe's priest declares he is "Neither god nor devil but a man!" (at which point one is brutally killed, and the other manages to flee). It could almost be read as an analogy for colonialism - the white men might have had a technological edge, and used shock and awe to take over, but as the natives catch on to what's happening, the risk of backlash and revolution grows.
Kipling wasn't firmly against colonialism, but he was a savvy (sometimes cynical) realist. Most colonials were pretty cynical about it.
Ever see the John Huston film adaption?
That story is absurd, considering that a lot of modern diplomacy is essentially deciding how much Dane-geld you should pay to appease America, Russia, or (insert your regional power here), and how much you could expect in return for promising that you will not pay the Dane-geld to the other side.
If you don't play, you end up like North Korea, ever so proud for their fierce independence, cut off from everyone else.
I find this cute tale, from a subject of the British Empire, doubly insulting. If you are powerful and you can extract Dane-geld from others, fine, but stop insinuating that other people pay Dane-geld because they're stupid.
And I say that as a Dane.
He asked me: do you know why we say "Skull" when we drink? I said: No I don't.
He Said: Well, back then during the Viking times. The Danes would fight all the way to the kingdom and cut the off the Skulls of the English princes; Dump out their brains and use the skull as cup for drinks. Hence the word "Skull".
I said: Huh? Interesting!
"And then we also took all their good looking women. And that's why all the Danish girls are soo good looking" Add him.
You got love the Danish people!
Back in the days people were drinking off bowls, instead of cups as we do today. And guess what the danish word for bowl is? "Skål".
A "skål" is a flat version of "trebolle". You can see one if you Google Image search for "trebolle" to get an idea of how it looks.
The Indians have always been a conquered people, it is only in the last 70 years that they have had freedom; you should thank the British for it.
India had been ruled by kings who were not originally from India. But, nothing in the history compared to the "loot" of the British (see http://www.theguardian.com/world/2015/mar/04/east-india-comp...)
I presume you are not the brightest bulb as far as Indian history is concerned so it will help a bit if you read up a bit on Indian history, even Wikipedia would be a good start.
See http://www.gwern.net/Terrorism%20is%20not%20about%20Terror#t... and http://www.gwern.net/Terrorism%20is%20not%20Effective