This is what a DDoS attack looks like | Hacker News Reader

This is what a DDoS attack looks like

geek.com

180 pointsetix70 comments

Disclaimer: VideoLAN president and lead VLC developer here.

The attack started on our new mirroring system (powered by mirrorbrain) 2.5 days ago, during the night (after 2am, so we were sleeping).

We were woken up (OP and I) in the morning by many mirrors complaining of high bandwidth use. The actual number of requests was not that high (400 req/s), but the botnet was downloading the whole vlc.exe, aka 22MB. So, we were at around 70Gbps during the night, in average.

Afterwards, North America got up, and things got worse. We had up to 1660 req/s, so around 292 Gbps...

This is very weird for a DDoS, to be honest.

Our front machine that splits the down mirrors was taking most of the load, and we were able to find the patterns to drop the botnet connexions, in order to not kill too much our mirrors. I won't discuss too much of the patterns, as you can imagine, but as usual, I'll be happy to discuss it IRL or by mail.

Tweaking the front server was also important to reduce the number of open connexions, to not kill our server.

2.5 days after, the attack is still going on, with an average of 500req/s.

The video was done using logstalgia, using scripts of OP, on my machine (<troll>he was running eclipse, he couldn't do both at the same time :)</troll>).

Is there a reason VLC doesn't offer a Bittorrent download option, by the way? As far as I know VLC's audience is pretty technologically-inclined, so they would understand what a magnet link is, and you'd be able to just switch off the archive-download option for a while in cases like this. (As a bonus, if you embed a magnet link in your page, you're implicitly embedding a SHA-verification of the resulting torrent file.)

"This is very weird for a DDoS, to be honest."

Maybe it was aimed at bandwidth costs?

Maybe. Seriously, no idea.

That's precisely what I don't get in this case, especially. The source is out there, there are other people mirroring vlc. So if you're trying to censor it, you're going to fail pretty hard. In the absence of that, what can it be?

Of course! It must be the mplayer folks!!

Or in this case, who you'd literally least suspect: wmplayer. All hail wmv!

I don't know what request throttling solutions are available for nginx. But, for apache there are modules and scripts/code that can rate limit traffic and requests from specific IPs. Depending on the size of the botnet attacking, rate limiting requests by IP or network may have been a way to mitigate the DDoS effects. Again, depending on the specifics, you should be able to use iptables (if you're running Linux) to limit requests per source IP or IP range. The type of actions you can take with iptables are really quite robust.

I've had to deploy request rate limiting solutions in the past because of well meaning federated search mash-ups that included content from servers that I run.

Filtering by useragent seems to be a temporary solution. I think that's especially true since you've disclosed publicly that's what you're doing.

With nginx you can easily rate-limit connections, requests and downloads (which you can throttle also). Can also do this by GeoIP - country, etc.

No need for extra steps - this module shall be compiled and part of nginx-full package from nginx official PPA, if OP is using Ubuntu.

I had problems with the Windows updater apparently downloading version 2.0.6 and then not installing it. I ended up downloading and installing it manually instead. Could that be related somehow?

VLC Windows updater?

Yes, the VLC Windows updater, which runs when VLC starts.

It has a very specific user agent, and does not call with this pattern.

Why do you return 403 (according to the article) instead of 444? 444 will drop and close the connection, so I am guessing it has something to do with keep alives vs connection overhead?

Who toes have you stepped on recently, I wonder?

Also, pattern-wise - most of the traffic in the video seems to be hitting a single .asc file.

No idea, seriously. VLC scammers, mainly.

The .asc is a logstalgia visualisation bug, because in fact, it merges the few .exe.asc requests with the massive .exe requests

What did you tweak in nginx?

Is this originating from specific country or all over?

All over.

"Disclaimer"?

Looks more like a "claimer" to me.

He wanted to write "disclosure" I guess. (He's not a native English speaker.)

https://code.google.com/p/logstalgia/

If you like logstalgia, you'll love gource[1] which visualises source control change history. It's also available as a brew recipe on the Mac for easy-peasy install.

[1] https://code.google.com/p/gource/

This has pretty limited use, but holy shit is it awesome. Thanks!

Limited use?

As someone who over the past decade regularly had to sell higher management on investing in infrastructure, I can tell you this will be extremely useful.

Seriously, even if it's just a random visualization of some normal peek traffic, it will go down easier than dozens of actual arguments, slideshows with bullet points or fat reports nobody reads. Show the video, do a bit of handwaving and there's your budget...

For any techie trying to convince their boss they need more servers, this is a very powerful tool. Use with caution.

So I installed the project and ran it with my log files and I'd like to retract my previous claim of it not being useful. You're right, I can definitely see the use here, it puts stuff into perpective quite well.

It's easy to write off shiny visualizations as silly (which is what I did above) but they definitely offer a perpective that you can't get any other way.

Thank you.

awesome find. thanks

Damn that's impressive. Someone please port a realtime WebGL version!

Love this

All I could think of was "this is what a DDoS attack will look like in Hollywood products for the next decade".

Awesome visualization.

I was just thinking, there weren't nearly enough laser sounds, polygons, or explosions.

It's not quiet there yet!

oh this isn't fantastic enough for CSI or BONES to use... not enough 3D graphics... very cool visual none the less..

Perhaps it's worth it to code a quick and dirty solution using JavaScript encryption. On your download page setup a script that would receive a given encrypted string, decrypt it with a provided key, and the use it to prepend to the download link. On the server, symbolically link the file on demand and send it to only one user, ip limited. This way the attack, though still can be automated, would require some code rewrite from that attacker, which might be beyond his/her abilities. Also, if the encryption algorithm is CPU intensive, then it wold require several seconds of CPU time per request from the attacker.

To make the decryption CPU intensive you may simply use any encryption algorithm you like, many are available as JS libraries, but instead of giving the entire decrypt key, skip the last 2 digits, and let the end user brut force the last 2 digits in the client via JS. That way there is a computational cost to each attack request.

Just some ideas off the top of my head. Not sure at the moment how to implement the server side part at the moment, but I am guessing that their are server side rules that allow you to easily set per ip access restrictions to folders or fils.

PS: please excuse spelling, typing this on my iPhone.

I'd be willing to go out on a limb and estimate that maybe some private interests in Hollywood, with certain four letter acronyms, despise open source media player projects like VLC, since they might represent a channel that can potentially enable bypasses that can circumvent precious, precious DRM.

The perception being: if you can see the source of a media player program, the encryption might be implicitly compromised. This is a silly idea though, because it neglects certain realities about the very nature of electronic encryption, and media consumption. Maybe having source code lowers the bar in some respects, but the reality is that determined people will simply bootleg media anyway, by other means.

Not an accusation though, just that my tinfoil hat is tingling. Who else might be so motivated to attack an awesome software project like VLC?

Cool / "good" projects get attacked all the time. I suspect this is because the attacker is guaranteed an audience on HN to marvel in the results of their work. I think this is probably simpler and more likely than the tinfoil hat theory.

Another option is some shady people bundling VLC with spyware, some of which have been successfully clamped down: http://blog.l0cal.com/2011/07/07/these-companies-that-mislea...

Or, less likely but more timely, the GPL→LGPL relicensing to get an AppStore port, or the consultation to get the French DRM police to play its role in enforcing a conflicting law about interoperability.

> Who else might be so motivated to attack an awesome software project like VLC?

Anyone who wants to show off a proof-of-concept attack to potential customers, without stepping on any huge toes. (e.g., it's relatively safe to DDoS VLC, not so much a U.S. government agency or a huge multi-national company.)

Why DDOS Reddit? Why DDOS Wikipedia? Why DDOS the New York Public Library website??

Within the last year or two I've given up trying to find reasons for DDOS attacks. I think of them now like Internet traffic weather. They just happen at random times and you better be prepared.

I could see it being some sort of message as CISPA was stalled in the Senate yesterday. More likely, however, is that we'll never know why.

I would suspect DDoS attacks to be akin to terrorist attacks in that somebody typically claims responsibility for them.

DDOS attacks are way more common than terrorist attacks, and there's a rarely a clue why they happen.

What kind of person would DDoS a video player website?

Maybe Videolan isn't the target and the DDoS is against someone else in the form of using Videolan's bandwidth against someone else?

That's my general assumption of any attack in which the response to a request is in the many MB range and the request is in the bytes range.

That someone with limited bandwidth and many connections is attempting to acquire a large amount of bandwidth to attack someone else.

Good thought, but source spoofing for an amplification attack wouldn't work here; from https://news.ycombinator.com/item?id=5613529 :

> The actual number of requests was not that high (400 req/s), but the botnet was downloading the whole vlc.exe, aka 22MB. So, we were at around 70Gbps during the night, in average.

Source spoofing would not allow downloading the whole file; the spoofed source address would get the first response packet and send an RST ("stop, no connection associated with this packet, go away") long before that point.

HTTP is over tcp though so it should be Hard to convince vlc's servers to reply to a third party. Seems like a silly ddos actually but what do I know.

The same kind of person who would steal books from a library, or talk in a movie theater.

It it possible this is an accidental DDoS? VLC is popular to bundle with things, and all it would take is the code that checks for a new version and automatically downloads to have a bug that it always thinks there's a new version...

This is impossible, seriously. I cannot share the details, but the HTTP headers shows clearly that this is not that case.

And more than 10 millions downloads per day?

Ubuntu downloads from it's own mirrors (hundreds of them), with very few exceptions for proprietary stuff that can't be distributed like Adobe Flash and ttf-mscorefonts-installer (not default). Also it would not download a .exe to install VLC.

Ubuntu 13.04 came out yesterday, the people upgrading might be reinstalling all there programs

Ubuntu does not download a .exe...

*their

I highly doubt 500 different people are upgrading ubuntu every second, but a bug in the software is possible I suppose.

And the bug just went away?

I actually have Logstalgia running with my primary server for Minotar, and at 4,000 requests per second this is normally what it looks like. Awesome program!

How does DDoS mitigation look like? Do they use realtime dashboard with similar visualization to cut off hotspots?

In my experience it is fairly manual. Here it is at a very high level. First you want to determine if this is really a DDoS or legitimate traffic.

You might be notified via downtime, alerts of load, in this instance, I suspect; download graphs, log analytic (if using a cdn which can handle the load, then you might not notice for a while i.e. eye popping bill).

Narrowing down the attack profile means looking at logs. Be that network flow data (very helpful) or in this instance web server logs. Probably something like: totals grouped by ip, destination url, etc to see if there are any spikes.

Also, managing stress. If you are some type of retailer then you likely are losing money, people are asking for updates, etc. This can be extremely stressful.

We use Munin for the graphs and Nagios for the alerts. The mitigation of the DDoS is done manually since we need to identify the patterns and block them.

Pretty cool stuff! glTail [0] does similar visual analysis of pretty much anything.

[0] http://www.fudgie.org/

I currently have a native OSX version og glTail in review by Apple, which is quite a bit faster and easier to install than the Ruby version. I also have an updated IOS version in the queue which doesn't crash quite as often.

An IOS version? How cool! I'll check it out, thanks for the heads up.

Fantastic! I had read about such a log visualisation tool a long time ago (I'm not sure but I think I read it about it via NTK which should date it) but I had lost any knowledge of what it might be until now.

Now I can see such a tool and it looks wonderful.

(More on topic, DDOS is beautiful!)

Brave of them to disclose it's just the user-agent they are filtering.

It's not possible to inspect the user-agent via the linux firewall (iptables) is it?

I guess you can use this if your iptables supports string matching

    --string "useragent"

I just got one of my servers attacking TicketMaster by a faulty cgi. (my alert system notified 5 minutes after it started) The mob is angry now It was disabled.. I think it has more targets that only vlc...

Is each request a unique IP or do you see frequent and recurring requests from a chunk of IPs?

Depends, I would say half of them are returning every once in a while and others only do a single request and never come back. No clear pattern here.

I guess someone took some VLC crash seriousley to the heart.... :D

Hang on guys!

I was browsing HN on a friend's computer that without adblock and clicked this link. Wow! Is this what the internet looks like without adblock? The ad/content ratio is crazy...

Without blogspam: https://youtu.be/hNjdBSoIa8k

Also follow the submitter link, currently etix seems to be most active on Twitter: https://twitter.com/etixxx

The VLC logo is a traffic cone.

That visualization sort of looks like a cone.

Coincidence?

Reply on news.ycombinator.com