Hacker News Reader

Comment by reacharavindh | Hacker News Reader

I wonder if there will come a cloud provider that solely uses consumer grade compute, local storage, and internal networking, with some kick ass network connectivity and steal the cloud provider show?!

Most applications that run on AWS and GCP don’t really need that premium Xeon/EPYC CPUs, enterprise-grade storage that is 10X the cost of consumer grade drives. Of course, those that do may justify the premium cost. But, the rest that just runs a few web servers behind a load balancer and non-critical services could use stuff on the cheap...

Say cheaper digitalocean built upon bare metal consumer grade hardware - Ryzen CPU, consumer grade memory, this Local NVMe storage, and the onboard network card... may be I am crazy to think that’s enough.

Heck such hardware may even perform better than several enterprise grade VMs provided by cloud providers.

View on HN

> I wonder if there will come a cloud provider that solely uses consumer grade compute, local storage, and internal networking, with some kick ass network connectivity and steal the cloud provider show?!

Forget consumer grade: even normal Dell/HP/Supermicro stuff is a PITA to run at datacenter scale. Most major cloud providers roll their own machines for a reason.

I'm guessing any hardware savings would be evaporated by increased labor, power, and footprint costs, likely making you more expensive than other providers.

> even normal Dell/HP/Supermicro stuff is a PITA to run at datacenter scale

not if you know what you are doing.

LOL.

Yes it is a pain. All these incumbent vendors have atrocious tools, management interfaces, and dealing with firmware is basically the same as 1990.

and in your estimation, 30 years (actually closer to 20) isn't enough time to get good at a thing?

“Knowing what you are doing” — yes, and what they are doing is throwing more labor, more machines and more electricity at the problem than either Google or Facebook ever would.

Major cloud providers got off “commodity enterprise” over a decade ago. There’s a reason for that.... the stuff is just bad.

With today's tech, it doesn't make sense. I'll try to explain:

> consumer grade compute

Non-ECC compute is a no go from start. You won't believe how computers make mistakes when they're driven to their limits. Even with good cooling, frying RAM, CPUs and ethernet gear is ordinary events.

> local storage

Consumer grade HDDs are slow. SAS is not consumer grade. Consumer grade SSDs cannot handle the write cycles. The ones which can handle neither cheap nor consumer grade.

> internal networking

Gigabit ethernet is cheap but, is not fast enough anymore. 10Gig+, Infiniband and other stuff needs reliable high bandwidth PCIe, which is not in abdunance in consumer grade hardware.

You need more lanes, then we come to next point.

> Most applications that run on AWS and GCP don’t really need that premium Xeon/EPYC CPUs

Server CPUs are not about processing power all the time. Reliability, PCIe lanes, expansion and advanced virtualization which allows that high performance network to be connected in abundance and shared between VMs without performance penalties. So you can scale the number of jobs running on less hardware in smaller spaces.

> enterprise-grade storage that is 10X the cost of consumer grade drives.

Again, it's not about performance but about reliability at least half the time. Performance is required to be able to scale, not for independent jobs.

> Non-ECC compute is a no go from start.

Many modern amd offerings support ecc memory, but I can't comment on intel.

> Consumer grade HDDs are slow.

Meanwhile cloud provider hdd storage is slower than usb2. I have a 32 core machine deployed in azure right now, and the hdd that was deployed with it has a max read speed of 10MB/s. Getting a medium performing SSD seems to cost in the realm of $250/month just for the storage, before computer costs.

For many uses, users don't care about true reliability. I don't care if my CI agent has data loss, I'd _much_ much rather it ran on consumer grade hardware at speed and failed 2-3% of the time than what I've got now (as long as I have the choice. I want my build servers to run on fast unreliable hardware, for example)

Agreed re: pcie though!

>Meanwhile cloud provider hdd storage is slower than usb2. I have a 32 core machine deployed in azure right now, and the hdd that was deployed with it has a max read speed of 10MB/s

That's because they're not bare hard drives directly connected to the server, they're a virtual hard drive over SAN.

Aha. Thank you. I'm not a sysadmin, and I couldn't figure out why I was provisioning machines and drives that had double digit integer multiples of the performance that is being advertise, paying heavily for it, only to be capped at such a small limit. Unfortunately my cloud provider doesn't seem to mention this fact _anywhere_ in their documentation.

> Meanwhile cloud provider hdd storage is slower than usb2. I have a 32 core machine deployed in azure right now, and the hdd that was deployed with it has a max read speed of 10MB/s. Getting a medium performing SSD seems to cost in the realm of $250/month just for the storage, before computer costs.

It's said before, but I want to expand a little bit. You're getting 10MB/sec because your volume is running as a virtual disk on a SAN system which runs on 100s on disks which can read 15+ GB/sec, but serving to 1000 servers at the same time.

If you had local storage, you'd have the same speed because your storage backbone would have the same throughput/consumer ratio.

You want faster drives or higher throughput/consumer ratio? You'd need faster RAID cards with faster drives and more PCIe lanes.

Guess what? This is enterprise hardware again :)

> For many uses, users don't care about true reliability.

Reliability does not always concern with noisy problems (data loss, crash, file loss, etc.). The biggest problem is silent corruption. A wrongly compiled, somewhat buggy binary. A slightly wrongly trained model. A wrong outcome from a computation or simulation.

Lost files are not a problem. Wrong outcomes looking right is.

I think OP's point is that many users aren't pushing machines to the limit or saturating 10g links. They just need the equivalent of a several-year-old desktop to run a webserver or whatever. Consumer-grade colo.

I completely understand that, but you concentrate these small loads to more powerful servers, so you can colocate ~50 small VMs to one gigantic server (in comparison).

When you squeeze 50 low-pressure VMs to a single VM, they become a one high-pressure load.

Hence, you need enterprise level hardware to be able to provide these low-cost, eh performance VMs to masses.

I deal with servers for HPC use. Let me explain why I made my question...

Consumer boards for Ryzen CPUs support ECC RAM these days. So, it's not as bad as you think. Is it as reliable as a server board? probably not.. But, I'm not building for or paying for that reliability either - fitness for purpose.

I think your comment on storage missed the whole point. The article was about a PCIe 4.0 SSD that I wished to use in servers. They are every bit as fast as the enterprise grade NVMe drives. The differentiation is typically power loss protection. At consumer grade service levels, I don't care, I go cheap, but insane value. Again, fitness for purpose.

Networking. I can find a Asus board for Ryzen CPUs with onboard 10G NICs.

Sure, server CPUs have more lanes etc and are designed for maximum load sharing across workloads. My quest is the opposite. Give me small, cheap, and insane value servers instead of VMs that are running dog slow in beauty servers used by a hundred others.

> I deal with servers for HPC use. Let me explain why I made my question...

Hello fellow HPC admin. I'm reporting in from another HPC center. Nice to chat with you. :)

> The differentiation is typically power loss protection. At consumer grade service levels, I don't care, I go cheap, but insane value. Again, fitness for purpose.

I don't think power loss protection is an enterprise level feature anymore. My Samsung 860 Pro has S.M.A.R.T attribute 235, which reads "POR_Recovery_Count". AFAIK, this attribute counts the times drive has lost power unexpectedly and had to do some housekeeping to recover itself.

AFAICS, the drive has 600x write endurance (1 TB version has 600TBW and it's linear with capacity), and if this thing is used in a VM server, it'd die in months. This is where enterprise drives matter. We have some SSDs designated as write caches for high performance ZFS appliances, and they didn't blink an eye to all this hammering.

> My quest is the opposite. Give me small, cheap, and insane value servers instead of VMs that are running dog slow in beauty servers used by a hundred others.

I understand, but as an HPC admin, you know that space is always at a premium. Having a Ryzen with 24GB ECC RAM and this drive on a non-system-room form factor is a waste of space and cooling capacity, and in today's integrated system room it's very hard to manage without IPMI.

So, at the end of the day, no cloud provider will touch this with a 10 feet pole. Even if they roll similar platform on a custom open-compute chassis, they'd opt for a more powerful system because it can a) can fill more roles, b) will be more efficient in terms of performance/watt since they can keep it more uniformly loaded and utilize it more.

From my point of view, this is the sad reality unfortunately.

Some providers are doing that. For example you can rent such bare metal servers from OVH's cheap brand So You Start. But that doesn't make sense if you need something more that one cheapest server. Consider the total cost of buying and hosting such servers compared to the resources they provide.

You are paying more but mainly when the capabilities are better. Example for SSD: Samsung 860 PRO 1TB and "for Data Center" Samsung PM883 1TB both have about 1200 TBW lifespan, both cost about $180.

On the other hand 8-core Xeon 4215 is almost 2 times more expensive than 8-core consumer CPU but it has 3x the memory bandwidth (6 channel controller vs 2 channel), 1TB of RAM per processor vs 128GB, 48 PCIe lanes vs 20 and supports 2-way SMP. That means you can put few times more capacity (cores, memory, in 1U rack enclosure than with consumer hardware. That additional capacity can also share 1 PSU, 1 network card.

I'm not even talking about ECC memory and remote management capabilities available only in server grade hardware.

ECC works on consumer motherboards nowadays, my X570 motherboard works really well with ECC.

Well the consumer grade stuff generally has higher clocks so things may be faster.

I think a lot of Devs have had the surprise that their laptop has higher single core performance than a server.

I don't know to what extent but space is certainly a premium. For a unit of rack space you want to maximise the cores and memory.

With a server grade multi socket CPU, you're also likely to pair it with at least 512gb of ram.

Consumer CPUs have a much lower limit. Though Threadripper is a bit higher.

There's also the availability side of things. How many CPUs can you get to populate your datacenter?

Will it be easy to order 500 consumer CPUs if you're not a retailer? Intel/AMD do want to steer their enterprise customers to higher volume parts.

Yes, there certainly are operational reasons to go big with non consumer-grade servers. In many cases, it is warranted. I was only imagining the lower end of infrastructure needs where a provider can get away with consumer grade hardware and achieve substantially lower costs. Say, a provider building their own racks of ITX motherboards with consumer CPUs. I’m sure there are enough Retail SKUs to be purchased in open market for small scale...

I think backblaze did just that with hard drives when they were starting out

Isn’t reliability a huge deal for servers thou? There must be stats on hardware failure rates where the cloud provider cpus should have significantly less failure rates than consumer grade ones, I’m going to guess that that’s gotta be a trade off. Consumer CPUs clock faster but the thermal controls are dependent on a single machine, where as the data center has to deal with packed server farms, thus sacrificing performance for long term thermal management.

I’m pretty sure Backblaze uses consumer grade hard drives.

I'm more thinking along the lines of arm server (low power) + VNMe storage + WAF = end of cloud needs for many current cloud consumers

Reply on news.ycombinator.com