How to destroy someone who hosts stuff at Hetzner dedicated server

It's worth putting this in context. Hetzner provides really beefy dedicated servers for ridiculously low prices [1].

You get great support (always had phone calls answered pretty much instantly and emails answered within a few minutes and all the techs I've dealt with knew what they were doing).

You can issue automated hardware resets and even get a remotely-controlled KVM attached to tweak the BIOS or regain access to your machine if you messed up the networking config (usually only takes a few minutes to get the KVM attached).

Orders for new hardware are also really fast - dealt with within the hour and often in under 15 minutes.

But there's no such thing as a free lunch. If you host at Hetnzer, you have to be aware of the reasons why they're so cheap, namely:

1) The servers are 100% unmanaged. They'll install new hardware for you if you ask them but everything else is up to you.

2) A lot of their hardware is desktop-grade, e.g. Intel Core i7 CPUs and non-ECC RAM. They do have some server-grade hardware in their high-end range however.

3) Their servers are in Germany. So you get quite a bit of latency if accessed from Asia or the West Coast of the US (see [2]).

4) They don't have any DDoS protection. In case of a DDoS, your server will get null-routed (but they tell you first). Again: 100% unmanaged. Up to you to deal with it. I've been lucky enough to not have to deal with a DDoS but my first port of call would probably be CloudFlare it it happened.

Provided that you're happy to do some sys admin, Hetzner is brilliant for a personal server, a CI server or even a prod server for a bootstrapped startup.

For literally next to nothing, you get a really powerful machine that will easily handle big traffic spikes without a breaking a sweat. And dedicated machine means that you get excellent and consistent CPU performance and disk I/O. If and when your startup takes off and you get funding, you can then choose between hiring a sys admin or moving to a more expensive host that offers a more managed setup.

[1] http://www.hetzner.de/en/hosting/produktmatrix/rootserver-pr...

[2] https://news.ycombinator.com/item?id=3898714

ovh.com and online.net provides even cheaper servers, but with a DDoS filter by default. I'm very happy with both.

I actually went to OVH a couple of days ago. I needed a new server and wanted to give them a try. But they're not accepting new clients anymore: http://www.ovh.co.uk/a1186.SoldOut

I didn't know about online.net. Shame I didn't see this a few days earlier. I would have given them a try.

OVH does not take new orders the usual way, but if you are buying as a company you can still get your order through by calling them or emailing them and saying how you plan to stay longer than, say, 3 months.

How long have you been a customer of OVH and online.net? How often were your servers under DDoS attacks? What type of content are you hosting?

I've been a customer for 6 years, and hosted various websites, including a Debian mirror and for a number of years, nedit.org. I don't know if any of these have been under DDoS attack at some point (though they can be quite high traffic), they "just work".

A lot of the people commenting don't seem to understand how hard it is to fend off such DDoS attacks. You either need some serious infrastructure (cloudflare style) or you need to buy equipment to mitigate attacks (like radware devices) or route it via a DDoS mitigation service (prolexic style). The one thing all these solutions have in common is that they are insanely expensive. People can buy a 1 gigabit DDoS for only a few bucks, whereas mitigating a 1 gigabit DDoS will cost you either $20K+ dollars for a mitigation device or some stupid amount of money to have a service like prolexic mitigate it for you. Services like cloudflare are a whole load cheaper but only provide basic reverse proxy protection and still leave your server vulnerable for attacks directed at it's IP instead of DNS name.

I can't say I've ever heard of Hetzner, but from the comments I'm reading they apparently offer servers for cheap. Bearing in mind how much money DDoS mitigation costs I don't see how they could handle this any other way without having to make some pretty serious investments (which in turn would make their hosting less cheap as the money has to come from somewhere, right?)

You can do some of it via BGP, which is a standard method for handling routing once you become any sort of server provider with multiple bandwidth providers. It is builtin to some Juniper devices already, for instance: http://njetwork.wordpress.com/2013/04/30/mitigating-ddos-att...

There are other ways to do it via BGP also. Plus there is null-routing, bandwidth limiting, etc.

juniper is the only one that provides that and flowspec is not going to be able to block everything. Other then that BGP is not really going to help with attacks.

IRCCloud had to move off hetzner for this reason. We were continually getting ddos'ed, and hetzner showed no interest in working with us to try and mitigate.

At one point they just suggested we "ask the responsible parties to stop", and closed the ticket.

Now we are on Black Lotus. Expensive, but the regular 50mb-10gbit ddos attacks are mitigated just fine.

...however, if you aren't concerned about ddos, I still recommend hetzner.

Excellent value for money dedicated servers, with good automated systems. You can remotely reboot a dedicated server into a recovery image and fix problems yourself. You can run the install process yourself too, so you get exactly what you want... except ddos mitigation.

Yes, and also except for the hard drives ... I got dedicated server which had hard disk problems after less than 1 month (both hard disks in raid were almost dead), and they offered me to PAY additional fee to put hard disks which is not new o_O (they claim it has less than 1000 hours operation, and asked me like 20-30 eur per HDD, I'm not sure honestly), or to give me another one for FREE but it was possibly also crap ... I asked them to put free one, and I had to reinstal everything and I had to manually backup all data, reinstall system and set everything up from scratch because the RAID was not an option with 2 "damaged" hard disks ...

Did the disks have SMART errors to begin with? They're a budget provider, you can't expect brand new hardware with every new installation. Though, I'd agree that if the drives had a high number of reallocated/offline sectors, CRC errors, and the like, then hetzner was at fault. However, given that the raid array initialized & ran fine for a month, it seems that it could have just been a case of disk failure.

As for making backups & reinstalling the system yourself. You should have expected that. Hetzner is not a managed provider.

That sounds awful, how long ago did this happen? I had extremely good support both times one of my drives failed, I just followed their tutorial on drive replacement and they plugged me the new drive. After that I let the RAID replicate to the new drive and was done. I'm from Germany so I don't know if the support is a different one for me, but they never asked me to pay for a replace.

That said, almost every (cheap) server hoster will have something in their TOS forbidding IRC use.

I have no particular insight into why IRC is so troubling, it used to be popular for malware botnets (C&C) and it attracts its fair share of script kiddies.

Wow, thank you so much for irccloud, it's amazing! Paying customer since about 5 minutes into the beta here :) I didn't realise you founded last.fm too!

Was that not also because of efnet klining hetzner?

that was just salt in the wound.. we were getting slammed by ddos and null routed on a regular basis before that happened.

Offtopic, but: IRCCloud is amazing.

Wow, you founded Last.fm and IRCCloud.

Thanks for both!

Yup, pretty much. Those attacks have become a real problem because they can be ordered so cheaply and easily that even kids use them in Minecraft feuds. The channel takeovers of the 21st century.

OVH's much more tolerant in that regard (ie. they keep your server online if battered) and all their servers now include a mandatory anti-ddos protection[1]. Unfortunately, they're fighting turn-over and don't accept new orders.

[1] http://forum.ovh.co.uk/showthread.php?t=6661

So I manage quite a few servers at Hetzner and we were DDOS'ed quite a few times. First, they warn you and if you don't get back to them in 12-24 hours, then they will shut down your server.

Sounds like you were unfortunate, but this is not generally what they do.

What would you do after they warn you? It's not really under your control to fix is it.

You can boot up some EC2 instances.

Help me understand...

What would those instances do, exactly?

Depending on the DDoS attack, they'd either help you load-balance or keep serving "real" users. That is, either the attack is on Hetzner only, in which case actual users would be redirected (e.g. you have multiple A records for your domain, some for Hetnzer, some for those AWS instances) to a working site. Or the attack is on everything related to you, in which case you'd utilize some load balancing to mitigate the volume of the attack - and depending on AWS' DDoS protection, the AWS part of your site might still be up, serving real users.

What would happen if I put for example 3xA records to 3 different IPs (in different locations), and one of them gets offline ... Does that mean all other traffic will go to remaining 2, or some of them will try to load the site from the offline IP ?

Web browsers will try the other A records if one of them fails, though it could potentially take them a while before they realize the host is offline. If the client receives an immediate error when trying to connect (such as connection refused or ICMP destination unreachable) the failover will be instantaneous. However, if packets to the downed host are just being dropped, the browser might sit there for 30 seconds waiting for a timeout before failing over. It's therefore best that you remove the downed host from the DNS as soon as possible (ideally from an automated monitoring process) and that the A records have reasonably short TTLs so the bad record doesn't remain cached for too long.

> Web browsers will try the other A records if one of them fails

No, they won't.

So load balancing via DNS records is the easiest way to deal with the problem, at the cost of (potentially) long failover times. What would be the next step to load balance your servers? I know Amazon offers elastic load balancing for their platform, but if I'm not using AWS or don't want to rely on them exclusively, then what would be my best course of action to load balance between 2-3 different VPS?

You should think of load balancing and geographic redundancy as two separate concepts, because they have two different best solutions.

For load balancing, the best solution is to put all of your servers at the same provider, in the same datacenter, and proxy connections through a load balancer. On AWS, you can use ELB. Outside of AWS you can roll your own load balancer using software like haproxy.

For geographic redundancy, DNS round robin is good, but I must emphasize that to do this properly you really need to have a short TTL and an automated monitoring process that removes downed servers from the round robin. If speed of failover concerns you, you can set your TTL really low (like 30 seconds) at the cost of slower DNS lookups. You have to strike the balance that suits you.

You can of course combine these - use DNS for redundancy between geographic locations, and at each location use a loud balancer. Note that each location needs to be able to handle more than its share of the traffic, not only because DNS round robins produce a very unpredictable load distribution, but also so a location doesn't get crushed under the load if another location goes down. Theoretically each location should be able to handle 100% of the traffic, but you can play the odds and skimp a little.

(Note: the very best way to do geographic failover is to get your own portable IP address allocation and an AS number, and use BGP to announce your IP address allocation from your active datacenter. If it goes down, you start announcing from your backup datacenter. However, only big players can afford to do this. For the rest of us, DNS-based failover is as good as it gets.)

http://en.wikipedia.org/wiki/Round-robin_DNS

I had that thing before (several months ago, also during DDoS), I received an email where they told me I have DDoS attack and if it continues, they will have to block my server. In this case however, I didn't get any warning email, just the one where they are telling me my dedicated is disconnected from network ...

That sucks. I have moved many websites recently from EC2 to Hetzner. what they offer is really impressive and the difference is clear (probably 5x more resources/power for 25% of the Amazon price).

I guess I will still keep the server, but will have to work on a quick migration/failover plan in case I encounter something similar.

I have also started using cloudflare as my default DNS host, so that could also be a possible solution.

Cloudflare doesn't help if they DDoS your server's IP directly ... You can also "hide" your IP by activating CF on all subdomains (the orange cloud thingy), but people always find a way to find server's IP and attack it (the CF doesn't help there at all, they only filter packets that are going through their servers which your domains resolve to).

How would they find the IP if you don't have it used in any DNS records? Unless Cloudflare exposes the real IP at times and you've taken all the proper preventative measures I don't see how this is possible...

There are ways. You could use services like domaintools and get IP history if you did use any of the IPs in the past. You could get the IP from e-mail headers, if the website sends e-mails during registration, password recovery, etc. You could look for ways for a server to make a request somewhere and log its IP, like posting an image on a forum, some forums do that. And this is just off the top of my head.

Right, in most cases though those holes are easily plugged. When switching over to CF don't use an IP that was ever public-facing for your site, use distributed systems like Amazon SES for sending email, etc. I imagine the things you mentioned do go overlooked by some when fighting off an attack, though.

What I would do is, just order a fresh new ip and point cloud flare to it. it is very difficult to find it and even the domain tools history will never know about it.

if the DDoser is really willing to invest more on attacking me, then my business shouldn't probably run on Hetzner :)

But then you can pull up another host and point to it. The original server is still DDoSed but clients can reach your website, which is what really matters.

cloudfare is awesome if all you need to serve is HTTP traffic. If you are serving something else, say, ftp or ssh or IMAP or something else? it doesn't help you much at all.

At least cloudflare will still serve your cacheable pages, right?

Here is a simple solution and everybody is happy: re-enable it every hour, if DDoS continues, disable again.

Everybody is probably "happy" then: Customer-> their unusable DDoSed server is disconnected, but wasn't reachable anyway. But once the DDoS is over, it's back online. Provider -> they have their traffic routed to null. However, they will have to do some more work to get this working too. And not to mention happier customers.

Here is a forum that sells DDoS attacks. Attacks are much cheaper than protection.

http://www.hackforums.net/forumdisplay.php?fid=232

You are kidding me right, nobody takes skidforms seriously.

http://quantumbooter.net/ this one actually works, that one is used by many kids, and even the one who attacked my server ... :S the prices are very cheap, and I've bought lowest packet to test it on my 1gbit server, opened ssh, started iftop to check for traffic and see ddos strength, activated ddos using SSYN type, and BOOM, connection time out from my dedi ...

I had to do deal with DDOS attacks in the past and DDOSArrest worked like a charm to mitigate the problem.

Link for convenience:

http://www.dosarrest.com/

BTW, does anyone know how what their prices are like?

(Their site doesn't have seem to have pricing info, just "Get a Free Quote" forms.)

How can DDoS mitigation devices distinguish between legit and malicious traffic? I'm not a networking expert, but it seems to me that if you're a website hosting a big file like the latest Ubuntu release, a legitimate client will say:

    GET /ubuntu-13.10-server-amd64.iso

and cost you 500 MB of traffic (or however big the ISO file is).

A DDoS is nothing more than thousands or millions of machines saying:

    GET /ubuntu-13.10-server-amd64.iso

How do the solutions others are talking about in this thread (DDoS mitigation provider or specialized hardware) tell the difference between DDoS traffic and legitimate requests?

That is something different, it is only used to waste bandwidth from someone (or potentially clogging server's upload, but its easy solvable), but in big DDoS attacks the attacker usually has several hundred thousands of zombies infected in his botnet, and then he orders all those zombies to spam packets at an IP he orders ... Every infected PC uses his maximum upload to target IP, resulting into something like this: http://d.pr/i/kmAn

If I'm online during the attack and check iptraf or tcpdump, I can see literally hundreds of different IPs spamming random stuff at me, completely overflowing my download until I get totally disconnected from server (time out), and I can do nothing about it, just watch it being offline ...

Wow, they detect the DDoS, but instead of blocking this they take off the servers?? Sounds ingenious..

Or are they unable to properly detect a DDoS and would also take off a server that hosts a web page mentioned on Hacker News?

How do other hosters handle this situation?

From what other people are saying, it sounds like Hetzner is the Walmart of service providers. You wouldn't see a traffic jam in the Walmart parking lot and then become indignant that they didn't have valet parking automatically start up to clear the parking lot traffic.

Rackspace basically do the same. I've had a Rackspace Cloud server null routed due to a plain vanilla SYN flood DoS attack (note the single D).

Detecting these is trivial. Actually blocking them requires significant bandwidth capacity and equipment. Do you expect them to make that investment? You have the choice between cheap hosting and DDOS protected hosting. Buying cheap hosting then complaining that your host is not providing expensive services is silly.

Use cloudflare or a similar service provider to mitigate such attacks?

Not everything on the Internet is a web server.

We had the same problem at Hetzner, the server was attacked on Saturday. We moved out. Hetzner is very cheap and you get what you pay for.

Yep, I would like to move out my files from Hetzner at this moment, but my server is locked, and I will have to wait Monday to get access to it ... Luckily I'm not hosting anything important on it and my business doesn't rely on them, or else I would be screwed very hard ...

You can unlock a IP address over the admin panel to get access to your server.

Great tip. Does anyone know who Hetzner's largest customers are? Or at least major web services that host with Hetzner?

They have a "notable customers" page: http://www.hetzner.de/en/hosting/unternehmen/referenzkunden

There are a couple NBA.com subdomains and Der Speigel appears to be hosting 'local' CDNable content for their Germany-based readers

Does this apply to servers that do NOT host websites? I host databases in Hetzner that aren't hosted in the same server as the website(they're in another provider)

In theory if they can find it, it can be DDOS'd. But not hosting public facing servers makes it much less likely to be a problem.

well this is good timing, just moved to hetzner last month and server mysteriously went awol yesterday until a reset...

Google this: hetzner hard disk failure They are putting faulty hard drives in their servers, and if you notice its faulty and tell them, they replace it with another faulty one (less faulty if you are lucky) ... Make regular backups to servers outside Hetzner network...

Consider somthing else than a ddos - attack. I realized probs with the hardware (RAM and Bios - Update was done in the middle of the night in 10min after telling them my insights)

FWIW, This is fairly standard.

Linode for example will null-route your linode for 24 hours if it's attacked.

It's quite irritating that hosting companies seem to see null-routing as a solution to a DDoS attack.

   > This is fairly standard.

True.

   > It's quite irritating that hosting companies
   > seem to see null-routing as a solution to 
   > a DDoS attack.

Not everyone can afford a proper solution. In fact, I don't think anyone, except OVH, is able to offer a server for less than $100/month and include proper DDoS protection.

If you don't need a beefy server and can manage with a mid-range VPS (or two), some VPS providers have <$100/mo offerings that come with reasonable attempts at DDoS management.

What alternative would you propose? With a virtual machine an attack on one instance can effect everyone on the same machine. Also actually blocking the attack is very expensive.

It's worse than that; a DDoS can also overwhelm the networking infrastructure, affecting other machines on the same switches, or in the same facility (depending on the magnitude of the attack, and the capacity of the networking).

Null routing does a good job of mitigating impact to other servers, but obviously causes problems. If there's enough capacity, filtering at the border would probably work. I think most of the attacks these days are DNS reflection (because it's easy and effective), so if the border routers could be configured to drop incoming udp from port 53 to the IP under attack, that would get you most of the way there (just make sure the server under attack doesn't need to get port 53 replies from the internet). That sounds simple, but it has three big problems: a) You need a lot of spare input bandwidth. b) You need to be able to filter on border routers. c) You need to be able to safely change the filters on the border routers.

DNS reflection, chargen reflection, and SNMP reflection. You can block them all fairly easily, but you still need enough upstream bandwidth to deal with them.

My alternative would be firstly to legislate that any network can only send packets that have a source address owned by them.

This would drastically cut down source address spoofing, which is the worst type of DDoS to try to cope with.

Secondly I'd setup a far better method of reporting and blocking traffic up the chain.

If they can detect the DDOS, they should be able to mitigate it, right?

(EDIT: of course Hetzner could choose to mitigate the DDOS by any number of methods - but they choose not to, because they have made a conscious decision based on cost.)

It's like a traffic jam. You can solve it by stopping the incoming cars at the ramp or by making the road wider, but both are outside of your control when you're just running the toll gates at some point ahead. You can only ask your "host" to do that (in this example, the owner of the road).

No. Detecting some types of DDOS attacks is pretty trivial. Just parse Netflow output and look for big bandwidth spikes. Actually blocking these requires significant effort to classify the attack to be able to block it without also blocking normal traffic.

when my dedicated was offline due to DDoS, I asked them to PAY for any kind of anti ddos protection, just to make my server online, and they refused that and told me they don't offer DDoS protection ...

I don't think they can detect DDoS attacks, just an unusually high PPS.

No.