EPYC Server Configuration | Hacker News Reader

EPYC Server Configuration

hetzner.com

102 pointsihsw254 comments

Note 512GB RAM costing 433.00 EUR a month (500 USD). Reminder: https://twitter.com/garybernhardt/status/600783770925420546

> Consulting service: you bring your big data problems to me, I say "your data set fits in RAM", you pay me $10,000 for saving you $500,000.

Previously, the cheapest such server was 700 USD at OVH and it only included half this much of NVMe SSD.

Also, I gave a talk years ago at NYCCamp trying to hammer in that most websites do not need all the scalability tricks as they fit into a single server. Well, this server can happily house a 7TB database on mirrored SSDs and that's still below 500 EUR (with "only" 128GB of RAM). You need to think very hard whether you need to shard, cluster etc.

You'll never get funding with that pitch, though. Hyper-modular microservice blockchain bus is what investors want to hear.

They read on the internet that the only way to survive FB levels of users is webscaling, and they won't make a bet unless they think you'll be FB sized in three years.

I am usually consulting for established companies with what they perceive as scalability problems but too often turns out to be a performance one.

That makes complete sense, tell me more. When you show them that there are enormous optimizations that they could be doing, what do they say?

Nothing. That's the end goal. My team is hired to make their website fast. Unless we recommend a total hosting change which is rare (but it exists) there's little involvement necessary from the client and it's usually in making the necessary compromises. Like, your posts will take two minutes to be searchable but in turn your search will become 400 times faster. Is this acceptable?

But what if you call it NoScale™?

Though it may be heretical to say so in the valley, you know it is possible to grow a business without funding, right?

In the valley you go can grow funding w/o having a business.

Your consulting advice will give you cold looks from those whose living depends on selling people complex server/"cloud" configurations under fancy names.

I am just channeling Gary Bernhardt who is the best. He also tweeted at one point that we will oil snakes into extinction.

well you need to pay at least 20€ per Server per Month (15 € Flexi-Pack 5€ second Nic) for a private LAN. (excluding the needed switch/switches for HA, I guess they are somwhere around 20-40€ per switch depending on if HA or not)

Isn't it cheaper to buy the server with that much RAM and colocate?

This one is under $2K:

https://www.ebay.com/itm/172727384582

Colocation is horror. You are responsible for replacing the HW if the magic smoke gets released. Nah. Just rent it. Not to mention capex vs opex.

Don't forget to add two 960GB U.2 NVMe drives. Those are expensive. For eg https://www.neweggbusiness.com/product/product.aspx?item=9b-... $831.60 apiece. Doubles the server price. Oh I forgot you picked a server which doesn't have hotswap NVMe bays...

Also that CPU is old and slow. It is the one before Sandy Bridge. I can't find the EPYC chip on the spec site but a roughly comparable Xeon:

Old: https://www.spec.org/cpu2006/results/res2011q2/cpu2006-20110...

New: https://www.spec.org/cpu2006/results/res2017q3/cpu2006-20170...

(yes, the latter is 2P so you'd need to halve it, it's still twice as fast)

I’ve colocated servers for a long time and even though it’s counterintuitive, the price of power+kvm+bandwidth+rackspace is rarely less than the price of a managed dedicated server in the same data center.

I'm a great fan of Hetzner services. It's not that their servers don't break - it's the way they react to hardware failures. Average response time is in minutes - they quickly replace the faulty equipment and everything gets back to normal. I'm using their services intensively and it always amaze me how quickly they can fix the problem.

I've had similar experiences with them.

Had a CPU cooler fail somehow (resulting in one very toasty, unstable server) and they responded within minutes, the whole thing was probably resolved within half an hour.

I've had exactly the opposite experience with Hetzner support. Slow to respond. Difficult to convince that hardware failure has occurred and never less than 3 hours, often more than 5 to get it replaced.

I'm guessing the difference comes down to failures they detect first, vs. failures you detect first. If they have a monitoring system that tells them something broke, they'll immediately go and fix it. If you tell them something broke, they'll have to go check. If you tell them something broke and they have a monitoring system that says it's fine, then you'll likely have problems.

If you are talking about dedicated servers: Those are unmanaged, Hetzner does not monitor those.

Regarding grandparent: What kind of hardware failure? I had to change different types of hardware in the past. For HDD changes you send a full SMART result + possibly the logs showing that the RAID kicked the faulty disk and they replace the faulty disk in 30 minutes (and one time even apologized that it took 2 hours for a low-priority-support server). In all cases the actual downtime was < 5 minutes, because of RAID. For possibly broken fans I simply stated that according to my Munin logs show consistent temperatures near the upper limit and asked whether they could take a look. They added an additional fan within 30 minutes, again with actual downtime < 5 minutes.

Most of my replacements were during the night in Germany, that might have helped, because there are possibly less requests in parallel. And I always had hard logs proving my HDD failures.

I'm very satisfied, too. Servers are ready in minutes, not hours, after ordering. We currently run ~30 servers there (dedi) and over the last ~7 years roughly had 1 HDD fail per year. Each time the replacement was done on our chosen time and quickly in about 5 - 20 minutes (the installabled images come with pre-configured software RAID 1). We recently tried a AX60 (Ryzen 7 1700X Octa-Core) and had some sporadic crashes, support changed some BIOS settings, no problems since. Service and prices are unbeatable, imho.

I know this is silly, but I would have laughed like crazy if your comment about using their reliable services ended with your comment stopping halfway (like your station/internet died).

Huge fan of Hetzner being so aggressive with its AMD Ryzen/EPYC chip adoption, but I can't ignore the fact that they had a huge data breach recently, and all customer info, including passwords, was in plaintext! I'm surprised nobody here mentioned it. Didn't Hetzner notify its customers about it?

https://techcentral.co.za/hetzner-hacked-customer-details-co...

https://techcentral.co.za/hetzner-deeply-distressed-data-bre...

Hetzner Online GmbH is a different entity than Hetzner (Pty) Ltd.

We've had our dev servers (heavy R&D work in machine learning) with Hetzner for 4 years now, never a glitch.

Super nice, responsive support too. I've had their Head of Product, Arno Pirner, engage in a long email exchange when we inquired about GPU servers last year (they finally launched a GPU line recently). Great service.

I just thought: could it be Radim Rehurek. It is.

I've had a wonderful experience with hetzner. The web UI isn't as slick as AWS/GCP/Azure and bandwidth is slow, but the actual machines with great and they're a fraction of the cost. For my workload, I got 20x (2000%) better price-performance than Google, which itself beat Amazon. Killer for R&D.

> but the actual machines with great and they're a fraction of the cost.

The advantage of cloud stuff is if you need an extremely high amount of power for an extremely short amount of time. For example, I needed a hundred CPU cores + TB of RAM for a 15-minute time span, paid less than 100$ - renting this or, god forbid, buying this, would have racked up 10k+ bills for sure, for buying probably over 200k.

Once you have base-line load that can't be served with micro/mini instances, there is no financial alternative to renting, and in some cases buying+colo may come in even cheaper (but requires HW maintenance personnel).

Snapchat, for example, pays 2 billion dollars for 5Y to Google, equivalent to 400M$ a year (per http://www.zdnet.com/article/snapchat-spending-2-billion-ove...) ... I wonder what their CFO (or any board member who approved this) has smoked. No way this huge amount of spend is justified - not even the taxing differences between capex (buying) and opex (cloud/renting) can justify this in any way.

In addition, all cloud providers charge heavily for data egress, which really makes me wonder how on earth a service like Netflix can actually be profitable.

Netflix operates its own hardware in ISP networks and at internet exchanges to actually deliver the video. The only thing running in the cloud is all the database and UI backend, but no big video files.

Netflix serves data from its OpenConnect rack-mounted servers, of which are deployed to every major ISP/IXP. Previously they were using mainstream CDNs but there were growing pains and cost concerns.

To sum it up, everything outside of "the play button" is on AWS. This includes billing, account management, et al.

This is probably a better link than the product purchase page:

https://www.hetzner.com/news/neuer-dedicated-root-server-ax1...

My biggest issue / bottleneck with Hetzner has been their network. 1Gbps is nothing these days if you run something larger. A mid-sized Elasticsearch cluster will eat this 1Gbps for breakfast during shard reallocation.

Other than than it has neat-perfect value per dollar.

You can order 10Gbps network cards and ask Hetzner to put your servers in a rack and a 10Gbps switch between them.

Pricing: https://wiki.hetzner.de/index.php/Root_Server_Hardware/en#Mi...

Excerpt:

> 10 Gbit Intel network card: € 13.50

> 12-Port 10 Gbit switch: € 43.00

I had massive issues with Hetzner the last time I tried a dedicated box with them (though that was 3ish years ago now).

In about a month I had a handful of days where I wasn't having issues, half the time I'd open a ticket, report a problem then a bit later get "yes, we are looking into that now".

Left a bad taste, these days I just use linode for most things, if I needed heavy duty processing power there are some decently priced options near to me (bytemark is about 60 miles away from me).

as a smallish datapoint, an i3.4xlarge costs $ 1059.94 ondemand

which is about 8 times as much, with nowhere near the throughput.

I assume with reserved one could knock it down to ~$650

It also comes with an unmetered 1 Gbit/sec port.

Assuming it's perfect, and you push it for a full month, AWS egress charges for that 328 terabytes: $20684.

It's 50TB/month, not unmetered. Only 1.17EUR/TB over that though.

Ahh, okay, so $4403 of AWS egress charges to match the 50TB included.

My experience with Hetzner was mixed. Great pricing, pretty quick to recover from problems, but the problems were frequent. They're not great about maintaining good availability. I would definitely consider using them again for any project where availability isn't a serious concern, but never anything customer-facing.

Epic indeed. I've been using their services for a few years now. Very good service all in all and very good value for the money. Thumbs up, Hetzner.

> 4 x 32 GB RAM

Because of their four dual-channel memory controllers, EPYC CPUs support eight-channel RAM. So, for optimum performance, it would need twice that number of modules, right? I don't know the performance differential.

No, they have 4 memory channels per Socket, and can use up to 2 DIMMs per channel. So provided only one DIMM is user per channel, this can fully utilize the meory bandwidth of a single socket. (It also depends on the memory module having enough ranks to provide enough internal parallelism, but the effect of that only shows in small memory accesses).

It's actually 8 memory channels per socket (4 times dual-channel)

Are you thinking of Threadripper? EPYC has twice as many dies.

Is it possible to buy EPYC workstations already? Components are quite scarce in the UK both for EPYC and Xeon Skylake.

This is great news for DDOSers! </end_irony>

80% of threats and DDOS attacks we receive come from Hetzner IPs! We stopped wasting time reporting this to Hetzner and simply block whole IP ranges in Cloudflare. Sad but true.

IP range blocking is extremely bad practice.

You'll lose customers that way.

It's like rejecting email addresses that don't end with .com.

There are lots of services these days that can "smartly" handle DoS attacks (e.g. only drop traffic from an IP in the range when it starts flooding you).

Sorry for my ignorance, but how likely it is that customer traffic originates from Hetzner servers?

Maybe someone runs VPN's trough them?

This is one possible explanation. For example, You'd be surprised how many people run intrusion attacks from Kaperskys VPN service....

You see Kaperskys IPS on your logs and wonder...

Range blocking is the last resort, and we've only done it after alerting Hetzner and getting no response.

In any case, we only block for a few hours.... and yes, you definitely risk losing clients. I know Hetzner is not only DE based and many user cases for IPs originating are possible, but luckily DE is not our market.

>80% of threats and DDOS attacks we receive come from Hetzner IPs!

Sorry, this doesn't sound believable in the slightest.

Ok, true. Some clarification is necessary.

We get hit everyday by 1-2 intrusion attacks as well as a few DDOS a month...and the intrusion attacks are mostly from anonymised address or singled-out ISP IPs...but of the ones that come from Cloud based organisations, 80% or more is Hetzner.

We've never seen attacks from AWS/Google Cloud/Rackspace, etc... but Hetzner shows up in our logs a lot.

> but of the ones that come from Cloud based organisations, 80% or more is Hetzner.

Probably because people renting servers from Hetzner, OVH and friends run them once and keep them running without upgrading the OS and software which means they will get pwned (and a pwned hexacore server with 1 or 10Gbit gives you pretty good resources compared to an AWS micro instance)... while AWS disincentivizes you from doing so. In addition AWS tries to enforce security with its heavy reliance on security rules, which adds another layer of security - after all no one can hack your mysqld when it's not exposed to the Internet...

Wouldn't cloudflare block these attacks automatically for you? I used cloudflare on a server that used to get dossed often and the attacks are now almost non existent when we got cloudflare in-front of us.

Range blocking is a really silly thing to do don't you think? Hope you guys find a better way that that. Something tells me your application might be leaking your site's IP?

Yes, sorry, should be more specific. DDOS are blocked, but intrusion attacks not so much.

Although cloudflare has also a Web application firewall, somethings always slip thru.

Re: range blocking, only very specifically and when it got out of hand...and just for a few hours. Not a good solution, I agree

Reply on news.ycombinator.com