A DDoS in Asia Pacific | Hacker News Reader

A DDoS in Asia Pacific

telegram.org

89 pointsboutcher45 comments

China is doing a mass arrestment*1 of 100+ human right lawyers last weekend, in the same times as DDoS start and end, and there's a news from China's official news agent indicate that Telegram is the main secret contacting tool that human right lawyers used.

Some people think it's China who attack Telegram, to avoid the lawyers to warning each other for the arrestment.

1) https://www.facebook.com/chrlcg/photos/a.1571958406350448.10...

2) http://news.xinhuanet.com/politics/2015-07/11/c_128010249.ht...

Interesting.

We know from this year's GitHub attack that (a) China is willing to DDoS foreign internet services, and (b) China is able to enlist foreign traffic to carry out the actual attack.

So it wouldn't be surprising if China DDoSed Telegram and made it look like the attack came from South Korea.

According to the founder [1], Telegram was even removed from Play Store for a few hours at the request of a South Korean competitor.

For whatever reason, somebody in South Korea is seriously pissed off with Telegram.

[1] https://twitter.com/durov/status/619486763032182784

LINE is a Japanese company IIRC.

It is a subsidiary of Naver, the largest (and politically well-connected) portal in Korea.

They've been having DDoS attacks since September last year, so it seems unlikely that it's caused by a recent event. I'm surprised they haven't done anything about it before now.

The attack in September occurred just as a large number of Koreans suddenly moved to Telegram. The mass exodus was triggered by a surveillance scandal where a major Korean competitor was found to be handing over a large amount of user data to the government.

In recent days, there's been another exodus of Koreans from domestic IM services due to the revelation that the Korean army has been a customer of Hacking Team, the Italian spyware vendor who got hacked last week.

The two incidents might be related.

Fucking assholes... Let the guys behind Telegram work, they're elevating the current level of messaging apps s big step further.

he didn't back anything he said. as much as i like Telegram, at least show some proof when you put such strong statement.

He just replied with more info on his claim: https://twitter.com/durov/status/620538556134653952

I knew it would be S.Korea. The company I used to work for, at the time I left, was dealing with some particularly spiteful individuals from S.Korea who have been DDoSing their gaming platform and their separate video host. This was happening off and on for about 12 months. Interestingly enough, each attack was committed by completely different individual and were unrelated. In one attack where the guy was caught (I think they caught all but 2 of the attackers), he claimed he ran the DDoS because he didn't like the fact that there was a Japanese pop music video being hosted on the video site. This wasn't a young kid either, the guy was 33 and had a full time job at some advertising company.

This more or less reflects my stance on SK too. I ran a reasonably large APAC/SEA esports-related site for a while, competing sites in SK often attempted to attack it (and outright told me as such, to get out of korea). It's strange and I really have no idea why.

We had a 9Gbps attack from SK earlier in the year. I have no idea why we were targeted, but my best guess is that a user in SK got upset at some user-generated content and decided to DDoS the site rather than report it to us. Weird.

Anyway, we decided to move to OVH, and haven't had any problem since. We did get an email about an attack being mitigated a few months ago (which didn't cause any outage at all), and since then the trolls have realised that we can't be DDoSed any more :)

There are assholes in every country.

It's easier for assholes in some countries to launch DDoS attacks than it is for assholes in other countries.

With the one of the world's largest userbases of outdated IE with tons of ActiveX plugins, South Korea sure is a nice place to run a botnet.

On the other hand, most of the ISPs mentioned in the article are not Korean, so maybe it's a bit more complicated.

Could this be a state-sponsored attack? Or an attack by nationalists who are against people bypassing the anti-everything-speech-related laws?

An attack sponsored by the South Korean government sounds unlikely. South Korea isn't exactly a bastion of free speech, but it isn't China, either.

If by "nationalists" you mean the notorious online community known as ilbe, that's definitely possible. They're a weird amalgam of political ideology and lulz, basically the neocon counterpart to /b/.

But it could just as well have been a shady competitor who got pissed off with Telegram for whatever reason. The social networking market in Korea is cutthroat. Almost everyone treats it as a zero-sum game where you have to destroy all the others in order to succeed. Maybe this competitor was planning to launch what it considered a killer feature and Telegram launched it first. Stickers?

Interesting, I hadn't heard of "iibe". Here's an article I found in case anyone else is interested: http://www.koreabang.com/2012/features/netizen-explains-root...

The linked post seems to point blame at LINE. Stickers (large animated emoticons) are a major source of revenue for LINE; if Telegram were to offer it for free then it will really hit their bottom line.

The majority of LINE users are in Japan and Taiwan. It's the biggest program in both of those countries. LINE is owned by a Korean company, though.

I got hit by two of these (1/27 to Feb 4th and 6/4 to 6/22), and they were relentless. It was difficult to know where the attack originated because many proxies were involved - most inside the USA). We only managed a 62% uptime during the whole affair, many customers were upset, and it really hurt business. We ended up refunding everyone for the month and sending out a huge apology, for which many customers were understanding. Still, it hurt our business dramatically.

This is most interesting...

  The garbage traffic came from about a hundred thousand
  infected servers, most noticeably, in LeaseWeb B.V.,
  Hetzner Online AG, PlusServer AG, NFOrce Entertainment
  BV, Amazon and Comcast networks. That said, the attack
  was distributed evenly across thousands of hosts and none
  contributed more than 5% of the total volume.

I used to host a lot with Hetzner, and while quite expensive, they mostly responded to these kinds of things very quickly and with a certain level of technical competence (which definitely cannot be said of every hoster). Also, I'm quite surprised to not see OVH in there, as their network has a kind of "reputation" for these things...

  Fighting back would‘ve been a little easier, if the abuse
  departments in most of the mentioned companies didn’t
  process requests 9-5, Mon-Fri only. (Hours more befitting
  a scuba-diving shop in Vatican.)

Business as usual I would say...although I don't scuba-dive...

Edit: formatting

Did you mean to say that they are quite UNexpensive? (because they are)

200Gbps (if true) seems very high for a non reflection attack.

This tsunami TCP SYN attack uses 1000 byte SYN packets apparently. A good countermeasure for these would be rejection of all large SYN packets. Verisign DDoS protection services claim that they can withstand 2Tbps attacks of most types.

Unfortunately this would break TCP Fast Open, which transmits data with the initial SYN.

I can tell you that almost no one uses TCP Fast Open. It's a draft RFC that violates other RFCs. Google has given up on it in favor of QUIC. You should give up on it, too. It's not going to happen. It's a bad idea cooked up by ivory tower researchers who have never run a network.

Would a client that supports TCP Fast Open then fallback to the standard 3-way handshake once it's SYNs timed out?

I'd normally say that doesn't seem that high for a botnet or collection of botnets. To put it in perspective, that's only twenty 10gig attached servers. Not that much when you think about it. Sure, you need transit to match the server but that's not uncommon at all these days.

The most unusual aspect of this attack was that it was an easily blocked, rudimentary attack using spoofed, big SYNs. Volumetric attacks have subsided and fallen out of favor over the past year. Everything now is layer 7 floods at high rates or low-and-slow to avoid detection. Either way it's mostly layer 7 these days. People I've talked with at Cloudflare and Prolexic have seen the same thing.

Also, we saw these big SYN floods about 3 years ago (before Radware coined the term). They are easy to block, the attackers went away, and we haven't really seen any since. I think this is a 3+ year old botnet run by an attacker who hasn't kept up with the times.

tl;dr this botnet is a bit long in the tooth

A lot of people have been claiming these types of numbers but can't really show the hard proof needed.

How would hard evidence be provided?

NetFlow logs would be one way.

Question: Is this possible because they are using Linux servers? The Linux kernel adopted TCP Fast Open?

https://www.ietf.org/mail-archive/web/tcpm/current/msg08204....

TCP Fast Open is a terrible idea, and RFC 7413 should be marked as historic so that no one actually tries to implement it. Google has given up on it in favor of QUIC and so should everyone else. QUIC allows for "zero RTT" requests that are also signed (preventing spoofing).

We started blocking these large requests over 3 years ago when we started seeing them. Interestingly enough, that was a full 6-9 months before Radware wrote an article and coined the term Tsunami SYN. We just called it "big SYN". The attack is trivially easy to stop, and anyone running a client that tries a TCP Fast Open should expect failure frequently.

Simple solution: move to OVH. Although they don't have servers in SE Asia, perhaps 100% uptime is more important than shaving 100ms off the ping time. (As far as I can tell they don't have real-time audio or video anyway).

What makes you think OVH could cope with a 200 Gbps DoS attack of this nature? A quick look at their services indicates they don't mention what kind of attacks they defend against, and SYN floods are some of the hardest to defend against.

SYN floods are pretty easy to defend against. Most devices do SYN cookies in hardware at line rate. We run midrange devices that have thwarted 40M PPS SYN floods without a problem. I'm not bragging or anything, just offering a datapoint that SYN floods are easy to stop.

They say they have facilities to clean 480Gbps of data, and 5Tbps of mostly spare inbound bandwidth, so their DDoS mitigation capacity is somewhere in that range (http://www.ovh.com/ca/en/a1171.protection-anti-ddos-service-...).

Customer testimony on places like Webhosting Talk have cast all of those numbers into serious doubt. OVH is more likely to nullroute your IPs than it is to fight off a 300gig attack.

Do you have a ref for that? I did quite a few searches and didn't find a single person on webhostingtalk saying they had been null-routed by OVH in the past year. Only one guy who worked for a competing hosting provider.

Shut down this guy's account during an attack:

http://www.webhostingtalk.com/showthread.php?t=1467534&highl...

That said, I'm not a personal search engine. Here's a link to search results:

http://www.webhostingtalk.com/search.php?searchid=1555010

He said OVH didn't give a reason for shutting him down, which seems unusual. Perhaps he's breaking their ToS?

OVH themselves say they protect 24/7 against DDoS attacks, regardless of duration or size.

480Gbps across all their datacenters. Each datacenter only has 160Gbps, and I doubt that they'll devote all of that to one client.

They do activate all 3 datacenters. https://www.ovh.com/ca/en/anti-ddos/hoovering-up.xml

OVH frequently has problems with false positives, or so some of their customers report. They may offer good value for non-critical services, but they aren't exactly known for 100% uptime.

I've been using them since March, and have had 100% uptime during that period. I haven't had any false positives, and the one time they did mitigate a DDoS I didn't even notice it. As I understand it you'll just have a slightly higher ping time when the DDoS is active. An occasional false positive with slightly higher ping time seems a better option than your site being guaranteed to get null-routed for 24 hours (as happened to me on iweb, although I managed to convince them to remove the null route after a few hours).

Reply on news.ycombinator.com