DDoS protection
wiki.hetzner.de
wiki.hetzner.de
The PX line is a Fujitsu Desktop PC with a server CPU and ECC Ram. For me that's a compromise I don't really like.
Also the network is a bit shitty since they don't route through DTAG (largest german carrier), you have to pay a premium so that they activate it for you, and that's only one thing that feels weird about Hetzner. Of course I understand the reasoning behind it, but still, why should I choose Hetzner if I can get a way better solution from Online.net?
On the other hand, if you need 24x 1 TB SSD storage with a 10gbit RPN connection: https://pbs.twimg.com/media/CblcJ2pUUAAtbhK.jpg :-)
mma@duvel:(~) mtr -rwc4 93.210.14.100
Start: Thu Sep 1 12:45:41 2016
HOST: duvel Loss% Snt Last Avg Best Wrst StDev
1.|-- 62.210.128.13 0.0% 4 0.3 0.3 0.3 0.4 0.0
2.|-- 195.154.1.20 0.0% 4 0.4 0.4 0.3 0.4 0.0
3.|-- 195.154.1.16 0.0% 4 0.8 0.8 0.7 0.9 0.0
4.|-- lag-online-1.dc3-1.rt.hopus.net 0.0% 4 0.4 0.4 0.4 0.5 0.0
5.|-- lag-pop-dc3-2.dc3-1.rt.hopus.net 0.0% 4 9.1 2.8 0.4 9.1 4.2
6.|-- lag-pop-std-1.dc3-1.rt.hopus.net 0.0% 4 0.7 0.9 0.7 1.5 0.0
7.|-- 62.159.61.37 0.0% 4 1.7 2.8 1.7 4.5 1.2
8.|-- 62.159.99.230 0.0% 4 22.1 22.0 22.0 22.1 0.0
9.|-- 87.186.202.209 0.0% 4 22.9 22.9 22.9 22.9 0.0
10.|-- p5DD20E64.dip0.t-ipconnect.de 0.0% 4 48.5 119.1 48.3 329.9 140.6
Because DTAG is a bunch of <insert choice swear word here>. What they did and continue to do to site/net ops, is amounting to extortion. It's good that Hetzner stands up to Telekom, lots of other net ops simply pay the extortion fee.
Obviously, these drives do not last very long, so you end up with hardware that fails more and more often as time passes. Makes you look bad in front of customers.
I guess it's a matter of pricing — Hetzner is relatively cheap, and you get what you pay for. But you should know what you're getting.
But I'll check out online.net, so thanks for that ;)
You can get more uptime (99.95%), more IPs, better and faster support with their business service (35 EUR / month).
The only thing that rubbed me the wrong way is that they downright refuse to delete your credit card from their system when you terminate your (fully paid) account.
But they are not the only ones (not that this makes it good)... Linode did the same to me, I asked kindly after their latest data breach and they refused to do so.
In that time I've had* one network issue < 1 hr and one instance where the server rebooted unexpectedly. Any other downtime has been either on purpose by me or accidentally by my hand. I can't even recall any scheduled maintenance that's taken my server offline.
* That I've noticed in my elsewhere-hosted monitoring.
My only complaint if I can even call it that is that the price of the server doesn't keep up with the specs they're selling - i.e. if you can get a server with <x> resources for cheaper than my server with <x-1> resources my price should come down to avoid me churning it). Having said that I don't know anywhere that actually does that and for reliability reasons it's probably better to cycle machines out when they get old enough for it to be worth the savings.
Also, depending on the nature of your application, you can often get away with that kind of downtime. Something trivial (yet extremely popular) like Twitter had lots of down-time and it made the users love the service even more.
http://www.theatlantic.com/technology/archive/2015/01/the-st...
I think the main lesson here is that your error messages are part of your UX and need to be treated as such.
What caught my eye is, if you scroll down long enough on dedicated servers page [1], you will eventually see GaaS - "Geek As A Service". This made me giggle far more than I am wiling to admit :)
Their support was responsive but ultimately unable to do anything.
Had a few machines on hetzner for years and rarely had issues. The few issues i had with them i got fast replies.
I did try online.net few years ago and had a lot of network issues (it wasn't stable, kinda like when you have a misconfigured vps and your neighbour is eating all the bandwidth at times). Have they improved on that?
How does OVH compare to the above two?
On the other hand their support is not great. I've heard it's all PC-grade hardware rather than server-grade. They complain Canonical tried to charge them out the backside for Ubuntu licensing when in reality it'd be nothing if they'd stop using a modified, unsupported and un-maintained Ubuntu derivative.
Hit or miss really, depends what you're after and what you're personally ok with.
VPS VC1 S - €2.99 /month
2 x86 cores 2 GB 200 Mbit/s 50 GB SSD
30 hours in a 24 hours day? :O
The more providers offer something like this, the merrier. I understand that this isn't a layer 7 solution, but that has it's downsides as well - Cloudflare (or any other reverse proxy) will MITM all your TLS traffic, for example.
It's also time to address the elephant in the room: AWS. "Oops, you got DDOS'ed? Here, have a $50k invoice"
CloudFlare also regularly speaks about attacks and mitigations, therefore is helping the community to build better defences. Other providers stay shy and never disclose their magic. We believe DDoS is an internet wide problem and one of the ways to solve it is to spread the mitigation know how.
Examples:
- DNS attacks https://www.youtube.com/watch?v=UcAygzNSxlI&t=2h13m20s
- Iptables is great https://www.youtube.com/watch?v=pCVTEx1ouyk
- Our DDoS mitigation pipeline https://www.youtube.com/watch?v=XiK4643YdOk
- BPF for DNS https://blog.cloudflare.com/introducing-the-bpf-tools/
- BPF for SYN https://blog.cloudflare.com/introducing-the-p0f-bpf-compiler...
- Kernel bypass with netmap https://blog.cloudflare.com/single-rx-queue-kernel-bypass-wi...
- NTP attacks https://blog.cloudflare.com/technical-details-behind-a-400gb...
- DNS amplification https://blog.cloudflare.com/deep-inside-a-dns-amplification-...
- Recent attack trends https://blog.cloudflare.com/a-winter-of-400gbps-weekend-ddos...
- L7 attack with ad networks https://blog.cloudflare.com/mobile-ad-networks-as-ddos-vecto...
Please do help us fix this. Report issues, help us understand when we have incorrect IP reputation. Help us find captcha accessibility problems. And maybe - join the team to actually code the fix.
We went through this in the 90s where a few people were upset that they couldn't send email directly from their dialup connection, because they were still thinking of the world as it was in 1993 before spam became so prevalent and anyone who ran a mail server was constantly trying to deal with thousands of dialup IPs trying to deliver spam. What actually worked was that people changed the way they worked to use things like authenticated SMTP relays so you could simply block entire dialup ranges rather than try to come up with a spam-blocking AI.
The browsing system could be improved by something like a CloudFlare login system or long-term persistent authentication storage so you'd only see a CAPTCHA once a week. Unfortunately, most of the complaints come from very pro-anonymity users – which is a legitimate position but also means that it's a community which is going to be significantly more likely than average to have things like cookie & JavaScript blocking, so the complaints would simply shift to “I shouldn't have to create an account!” or “Why can't I disable JavaScript and cookies for your site!?!”
CloudFlare (and everyone else for that matter) should stop assigning "reputation" to IP addresses. An IP address is a network location, think of it as a temporary storage box which could be occupied by anyone and anything. We should move beyond coloring boxes.
Also, I think the captcha solution is better than what most administrators do, which is to block those IPs outright.
Cloudflare's job is to block potential attacks, and blocking or CAPTCHA-restricting a subnet tied to a large number of attacks against their infrastructure seems reasonable.
Either switch VPN providers, or deal with the CAPTCHAs.
* Or any other reason you're using it.
Edit: If I'm wrong can you please reply explaining why? I'm having an emotional rollercoaster of up and downvotes here! Please do feel free to correct me if I'm wrong, we all need learnin'
There's not much to be done about this otherwise - a Tor user is sharing a network with a significantly higher than usual amount of the bad elements of the internet.
Oh and don’t forget they’ll even put the captchas on subdomains, like img.domain.tld. Go visit stackoverflow via VPN/TOR and watch how the site has no styling/images even if you do their stupid captcha on stackoverflow.com. They’ll still serve you another one on static{1-50}.stackoverflow.com. Fun.
There's a very good reason why Cloudflare does this. Cloudflare isn't "punishing" anybody, their job is to protect the people that use their service. Tor (or any other open proxy) is a massive source of bogus and/or abusive traffic.
It wouldn’t be so bad, except Cloudflare doesn’t have any idea what they’re doing. Like the static.domain.tld issue I’ve mentioned—not even documented. This breaks websites in a very, very, bad way.
Next we have the lovely security setting called “Essentially-off.” Which does absolutely nothing to the captcha system. Apparently “essentially” means “not really,” according to Cloudflare.
Also forget VPN/Tor. I’m getting consistently captchad by Cloudflare just visiting mostly american-only websites from residential networks in Europe.
They’re either incompetent or just plain malicious. Right now I’m swaying towards the latter :/
Nobody is saying "hey this guy has privacy, lets make this experience a pain in the arse". They're saying "Man this set of specific IPs are really hammering my system looking for WordPress exploits and we're not even running that".
To treat Tor better than regular traffic would be to discriminate against Tor, which would invite more abuse of Tor.
Man just read up I'm essentially just summarising this anyway: https://blog.cloudflare.com/the-trouble-with-tor/
It makes perfect sense to add additional verification challenges for Tor users.
Also, anecdotal, but I work in infosec for a large US company and see a lot of traffic and read a lot of logs every day. The majority of all spam, fraud, and scan activity comes from Tor, or consumer-advertised VPN services, or various European VPSs. Residential ISPs (excluding Chinese and Russian ones) make up almost no part of it, relative to those others.
I'm pretty sure Cloudflare's challenge criteria is sourced from real data they've generated from traffic they've seen. They aren't trying to single out Tor users.
I’m actually perfectly fine with that. The issue is how Cloudflare does it. They should just offer something like X-Cloudflare-Fraud-Score header or even <x-cloudflare-protected-frame /> tag (they’re already doing SSI), but instead they just nuke the hell out of everything.
The way they do ‘protection’ is openly hostile to open web, or even neutral web. Whether this affects Tor disproportionately or not is a minor point.
I like Tor as a concept, the problem is that there is no way to stop people massively abusing Tor.
Whatever you want - nazi/hate speech, swatting, trolling, DDoS (by hitting expensive render paths or by forcing cache bypasses) - operate any kind of site with user interactions and you will get messed around with by mostly Tor-using scum, since trolls have understood by now that exposing their real IP will lead to them being v&d.
I don't like Cloudflare as a SPOF for half the internet, but for now they seem to be the only one able to reduce the impact of crap you have to deal with as a site op to a manageable level.
Trolls are destroying the Internet.
I'm very glad CF exists. It certainly doesn't catch all bad traffic in my case either, but it helps. I don't think any cloud security service can ever stop all spam or bot activity.
{blinking text|viruses|spam|IE6|blackouts|myspace|visual basic|2400bps|java|javascript|EMP}
rumors of its imminent demise might be exaggerated.. ;)
With the amount of sites they have behind them, and the amount of visibility and data they can extract, it is actually a bit scary to think what will happen if they get compromised or "do evil".
But so far, they seem to take the open and ethical approach and actually follow the "do no evil" mantra. CloudFlare is another Google in the making and I hope they keep like that.
thanks,
So, thank you!
Negotiating the bill afterwards is an experience most people want to avoid, I think. I guess many would prefer a service where they don't receive the huge bill to begin with, even with other downsides.
When a service provider decrypts and filters traffic for you, it's usually just called decrypting and filtering traffic.
Stripping encryption implies that there was, at some point, some sort of encryption between the browser and the website itself, which there never ever was. In the Cloudflare architecture the browser never ever interacts with the website, always with the Cloudflare servers. When you've got the small lock, it never implied that you were connecting to the website, but always that you were connecting to what lies behind the domain name, which is the Cloudflare servers. As an additional point, the website owner has always agreed to use this architecture; they're not somehow victim of some kind of attack from a spooky middleman. They ask (heck, some of them even pay) for it. It is part of the way they want to serve content.
You should tone down the sentimental analysis and keep to the facts. Cloudflare is a reverse proxy service the website owner uses; it is part of their infrastructure.
Indeed.
It's almost as if metaphorically speaking they were standing there between them... like a man... in the middle...
So, why do people call CloudFlare MITM?
1) They honestly don't know that MITM is just the name for an attack, not a neutral term.
2) They think any mid-route decryption/re-encryption is bad and dangerous.
3) They are mad about some aspect of CloudFlare's service, and call it MITM to make it sound worse to everyone else.
Folks in the 1 and 2 camp just need to get educated. Folks in the 3 camp might have good points on the substance, but they are contributing to the misunderstandings of 1 and 2.
Reality: Someone else (Cloudflare) claims to be the name on the certificate, whereas in reality they’re not.
That’s the textbook definition of MITM attack, whether you like it or not.
Of course all CDNs work like that. No one’s claiming otherwise. Cloudflare is just bringing all the ways internet is fundamentally broken to the forefront. Their unapologetic and extensive abuse of those design flaws (captcha pages) are a constant reminder of just how a hopeless mess the internet is.
Calling Cloudflare MITM but not Amazon is just demonstrating that you haven't thought carefully about how professional service providers work together to deliver a website.
The certificate authenticates that you are seeing the website that the website owner intended you to see. If the website owner hires Cloudflare as a professional service provider, that premise is not violated!
Anyway as I stated before the bar for calling it MITM is simply the intended behaviour of the protocol (HTTPS). Which is very much violated. Whatever negative or otherwise connotations the term has doesn’t change the fact that it is technically correct. Really all I’m arguing here. There’s plenty of better reasons to hate Cloudflare anyway.
This puts you into segment #3 in my list above. And I agree with you: the CloudFlare "flexible" SSL that sends last-leg traffic in the clear over the open Internet is a terrible idea.
But while that terrible idea opens up the potential for MITM attacks, it is not, itself, a MITM because the website owner has authorized CloudFlare to do that.
And of course there are levels of CloudFlare service that do properly re-encrypt the last leg to the origin.
> Anyway as I stated before the bar for calling it MITM is simply the intended behaviour of the protocol (HTTPS). Which is very much violated. Whatever negative or otherwise connotations the term has doesn’t change the fact that it is technically correct. Really all I’m arguing here.
You need to understand that you are not correct on the technicality. None of the protocols used in HTTPS are violated by decrypting or encrypting a session using the proper keys, even if it happens several times along the way.
Not every bad idea is a technical protocol violation. Not every bad idea is equivalent to an attack.
Abusing the term "MITM" makes it harder to talk with specificity about actual MITM risks. If CloudFlare itself is always a "MITM", then how do we explain why their Flexible SSL service is riskier than their Full/Strict SSL service? The former makes it easier for some mid-point to impersonate or alter the origin website. But now what do we call that? An "actual" MITM? A "real" MITM?
Let's just leave MITM to mean an attack. It's a lot clearer that way.
The term MITM is appropriate because it shows the whole thing is a charade. If handing off secret keys to third parties with absolutely no transparency past the first hop is the standard modus operandi then the entire protocol is a big fat joke. The only purpose, as far as I can tell, is to further centralize the web. First it was ICANN with domain registration, now it’s certificate authorities. Soon you’ll need a special permission from government to host a website.
HTTP(s) needs to be burned to the ground and replaced with something better. Liberty and freedom are at stake. How’s that for FUD.
The only issue with both is that they don't handle l7 DDoS, which seems to be getting more common. I also don't like that they leverage TCP rst's for syn floods, but I guess thats better than going down.
But so far, for l7 attacks you still need ddos mitigation strategies or something like CloudFlare.com or https://sucuri.net in front of your site.
thanks,
That just creates a wrong impression.
AWS charges a whooping 90$ per TB, it keeps me wondering why their traffic is SO much more expensive than Hetzner's...
With places like Hetzner you may be shut down for abusing their services if you use that limit too frequently.
(Not on a recurring basis anyways -- yes, there's the one-time capital costs and such.)
And all that is especially true in the case of Hetzner, who have repeatedly demonstrated how to build hilariously broken networks. Even ignoring that they were vulnerable to ARP spoofing for a long time and other things that have since been fixed, just some blunders of their current offerings: virtual servers behind v4 NAT (yes, you can't make that shit up ...) and IPv6 assignments of a single /64, even for dedicated machines (with an option to extend it to a /56 for money). So, if their new blackbox screws something up, what are the chances that they will care?
What sort of scenario would benefit from the announced DDoS protection? People are surely not running websites on commodity hardware.
I'm trying to figure out if you were expecting support for things that most dedicated, colo and cloud providers wouldn't support (OS updates, install nginx, ....) or if these were clearly provider issues (hardware and network).
> People are surely not running websites on commodity hardware.
The most recent issue we had with Hetzner was when we requested a LARA remote console to connect to a failing machine. The standard keyboard on the remote is German and much of the non-Latin keys are remapped. It was impossible to log in to a shell. All we got from support was "This is a standard Generic 104 key layout". That was it.
Before that Hetzner rebooted a whole raft of machines and on one of them this corrupted ext2. We asked what prompted the reboot (not bothered by the dead server) and we got (as above) "Unfortunately we can not help you here".
There were many more incidents before that and at the end it was just too much to swallow so we moved away.
Just out of interest, what's your use case for Hetzner?
A central location to take writes. Writes to this location go into a database and a durable, at-least-once, queue (writing to the queue is as important as writing to the database).
Then you put your API servers in different geographies (and preferably different hosting vendors). They listen to the queue and update their own storage (which could itself be a full relational database).
Edges aren't caching layer, since all the data is expected to be there (so, no miss). But certainly some of the data could be loaded in this way. The edges can also take some writes and send it back to "central", but you have to not care too much about the consistency (and maybe even accuracy) of that specific data.
Throw a latency-aware DNS on top of it (anycast, geo) with a short TTL and good health checks and you don't care too too much about uptime, allowing you to focus on raw price and performance.
Ironically, the only time I remember a serious outage (at a single location) was during a DDoS against a specific location at a much more expensive provider (not-rackspace-but-the-other-one-you're-thinking-about) which, of course, null routed us.
That actually is such a growing trend that one of the more popular NoSQL databases is called...
Cluster Of Unrealiable Commodity Hardware Database,
aka CouchDB.
If this has kicked it, I guess that might explain it.
Funnily I saw a thread on the customers forum at forum.hetzner.de today about why Hetzner does not have a better reputation social-media wise. Seems not to be on their priority list. There is even an English subforum, you may try to register there and ask for experience by other customers.
There is also the 'German' view of customer service to consider (i.e. the customer is not always right).
They target their offer a bit differently than other providers and this puts off some people, who don't appreciate the tradeoffs.
The details and tradeoffs are a bit too many to list here, but basically when they can cut corners to lower prices, while still keeping server-grade performance and uptime, they do not hesitate to do it.
It's also a bare metal / every server for itself / full DIY offering, without expensive gear such as SAN, blade servers and such. So if you're not prepared to roll your own virtualization, failover, backup, provisioning, etc solutions, you are better off going somewhere else.
That being said, the network service and server specs you get per your buck are second to none. The support is also adequate, if you understand the limitations of their offering.
Source: been a customer for 10+ years, currently have 10 servers with LXC and ZFS-on-Linux, hosting 100s of websites, mail domains, and custom applications, including a full backup solution based on ZFS send / recv. Would cost me a fortune on any other provider.
They'll answer competently. I sent a spam complain there once and got a sensible reply from the CEO.
But if you buy a product where they supply hardware and you do all the root work yourself, and then ask them to help you recover after a file system problem, they'll brush you off and you might consider their response rude.
Edit: be aware this is a bare metal server. You have to do everything yourself.(monitoring, raid integrity, smartd). And do not forget to request a remote KVM console(LARA), if you plan to reboot and you are not sure the kernel will boot.
Caveat: (Root servers) They won't help you find problems or solve problems. They don't monitor your hardware. If you've got hacked and send spam, they close you down. If you have harddrive problems, you need to show them otherwise they will not easily swap drives. It's bare bone, though if they are responsible for the problems, their support is fast when notified. They are also helpful, e.g. when moving servers to a different data center, keep old servers so you can migrate to new ones etc.
What these companies need are just additional services and nice management tools which make the whole dedicated server and network stuff look more like what we see on AWS or Azure.
My assumption is that at some point more people will come to the conclusion that part of their workloads such actually run on cheap dedicated hardware. On Hetzner €60/month buys you 4 cores, 32GB, 480GB SSD, 30TB traffic. Compare that to Azure A2, which at €64/month gives you 2 cores, 3.5GB, 60GB.