New x86 8U servers with 32TB RAM, 480 physical cores, 15 PCIe x16 slots, 400GbE | Hacker News Reader

New x86 8U servers with 32TB RAM, 480 physical cores, 15 PCIe x16 slots, 400GbE

lenovopress.lenovo.com

53 pointsMichaelZuo56 comments

There are also similar products being introduced by HPE, Dell, Supermicro, etc...

Posted here since it's pretty surprising how far commodity x86 servers have come.

For example, there are very few scenarios I can think of where even large startups would need more then 32TB of DDR5-4800 RAM or more then a few 400G Ethernet links on a single system.

As the marketing states, this isn't targeted to startups of any size, or any other organization capable of writing software. It has exactly one application: SAP HANA. SAP is a large and valuable company with many users of their terrible program, reputed to be among the worst programs ever perpetrated, which is nevertheless quite widespread.

It's very likely there will be some customers not running HANA shops.

But anyways, whomever buys them, it probably has a small overlap with the average HN reader. So it's meant as an illustrative example in a familiar context.

EDIT: Then again it's possible someone might buy it for personal use, e.g. if someone has multiple terabytes of spreadsheets that they need to work on simultaneously.

I'd buy one just for bragging rights. Why not have 32TB?

I can think of many ways, use it as a ramdisk NAS, minecraft game servers..

That's a good point. Also, 32TB of memory is just barely enough for customers who want to surf online with half a million chrome tabs open.

You should see Teams.

Nothing like running a single threaded game on 480 cores.

To run your game, Windows has to run about 200 instances of svchost.exe.

> many users of their terrible program, reputed to be among the worst programs ever perpetrated

And yet, no one has produced something better. Everyone complains Concur is dogshit but no one has presented a worthy competitor. Where is the "SAP killer" startup? I'm sure someone will publish one to GitHub any day now.

Just like Threads, the "Twitter killer", dead on arrival.

Pretty much every expense system I've ever used was better than Concur. When the business is using SAP for everything else it doesn't really matter what minor stuff like that is best of breed though.

I work with internal systems for a living. As an end user, you’re not the primary customer - finance is. Concur is by no stretch amazing, but it works fairly well on the back end where your expenses come in, route for approval, get paid, get accounted for, etc.

I’m not trying to be dismissive. I absolutely try for the best internal user experience across the board, and it’s led me to try a lot of the shinier startup T&E systems. But Concur works well for finance and accounting, and frankly, it’s also cheap - something you also have to factor in G&A tooling. Those two factors together have overcome my desire to rip it out at a couple of different employers.

And given the price sensitivity for a tool that does not generate revenue for its customers, it remains a tough space for new entrants to disrupt. It’s not enough to make it good, you also gotta make it cheap. When I’m done with the startup carousel, I might try my hand at this - but you should bet against me!

I made a SAP killer in lotus Notes. Ok maybe not exactly, but at IBM the choice was to wait for SAP to come in and do the project management with that, or have me (intern) port existing non Y2K compliant green screen project management program to Lotus Notes. It wouldn’t be as feature rich as SAP but it would do what was needed.

Because I cost many orders of magnitude less and my Boss kinda figured rightly the SAP project would be really late, they ended up with my database application which the users liked because I added features to make their lives easier.

Custom tailored apps really can be an amazing thing. I wonder why more companies never really got on board with the Notes/FileMaker databases. My lab currently has a pretty custom management system that tracks everything, prints labels and just really is a force multiplier.

A huge number of companies developed custom internal Lotus Notes (IBM Domino) applications. It just wasn't widely publicized.

Notes was great for quick and dirty workflow applications. You could literally build and deploy some simple functionality in minutes. But it also made certain more advanced features effectively impossible to build at all; it just wasn't a general purpose platform. And the fact that source code was locked up inside binary files created huge obstacles to working in larger teams.

Nothing is more exemplary of Oracle's incompetence than the #1 database company being unable to produce competitive ERP systems that are nothing but databases with front ends.

ERP is also something that should technically be achievable with a large industry-sponsored open source project but it never happens.

This completely misses what the SAP business actually entails. No one runs "SAP". They run "SAP as customized for business X". That customization costs millions of dollars. It encodes every little specific thing about the business. Companies run their entire operation on top of it. It's impossible to replace, because it can't go down. Ever.

An open source version would be only step one, and not that useful by itself. You'd still need to create the entire consultant ecosystem SAP has on top.

I'd pay for HN premium to double upvote this.

> ERP systems that are nothing but databases with front ends.

I’d love to see the face of anyone operating under this belief when they get to the tax code of any country that is not the US.

God damnit, SAP... Somehow I forgot about them. Having been exposed to their Job for five weeks 16 years ago, I can attedt that they are horrible.

> is a large and valuable company with many users of their terrible program, reputed to be among the worst programs ever perpetrated, which is nevertheless quite widespread.

A very good description of Microsoft or Google (and Apple getting there soon).

> It has exactly one application: SAP HANA

it says "and SAP HANA", there are lots of database cases for system like that.

I need one of these machines for Yocto. Actually, maybe two of them.

I love the symbols on the cross connects to turn the two 4U into an 8U.

CPUs: https://en.wikipedia.org/wiki/Sapphire_Rapids

With 32TB of ram it looks appealing to not even bother with a RDBMS and just use a full text database and keep everything in memory lol.

It's not Big Data until it's over 32TB now! (or 64TB if your a Power10 user)

You laugh but I have done some crazy speedups via ramdisk techniques.

Early Minecraft for one.

If your data is mutable, as in most situations where one uses a database, you still have to hit the disk for writes and wait for a disk sync before continuing after the writes.

Reads are indeed fast, but a proper RDBMS will instantly use all of that memory capacity for caching anyway, meaning that you get the same read speeds even with the RDBMS. No need to change your code.

There are certain algorithms that are more used on in memory data structures that could be leveraged and never touch disk, not just simply boosting a relational database.

That being said my post was somewhat in jest

If all that 15 PCIe x16 could be used for CXL at PCIe 5.0 speeds, that'd be just shy of 1TBps of attached memory, and one could hypothetically scale out to whatever size one wanted too.

Samsung dropped a 256GB CXL memory drive quite recently. These numbers should keep climbing and climbing. Very early days still yes. Sure does seem like in-memory will be enough for most users soon.

And CXL 3.0 introduces switching. So multiple servers can connect to the same memory device. Tons of interesting options to revise how things work as that starts to happen. The conventional ideas of networking are kind of silly when you can just have some other computer access the same memory.

RAM size > data size has been the case for most DBs for a while, software has been slow in adapting to it.

Also 12 power supplies and 24 fans. That's up to 96 amps @220V or 21KW, although I imagine it draws less than half that since the power supplies are n+n redundant. If you were thinking about getting one for home, a circuit for a typical electric drier or range wouldn't even be rated for this.

> Also 12 power supplies

Connected to the same wall socket ?

I have a Dell R960 with 96 sticks of 64gb ram which sums up to 6TB. But damn would I love to have one of these, not only for the much higher capacity but also the higher memory bandwidth from ddr5.

Kinda of crazy, my first computer had a million times less ram than this.

> Kinda of crazy, my first computer had a million times less ram than this.

My first computer had a billion times less RAM than this: not even 32 KB (16 KB actually). My everyday computer has 2 million times the RAM of my first computer (16 KB vs 32 GB).

And now we're waiting for the real dinosaurs here to comment!

My first computer had RAM.

What gets me is how it has made all my intuitions about how to program incorrect with time. And I'm no dinosaur. I started with an old obsolete machine with 128 KB. I remember getting a hand-me-down PC with a 486 (virtual memory at last!) and 16 megabytes of RAM and I couldn't imagine how I could ever use that much memory, besides for multimedia. Still hard to imagine, sometimes. How long to sort a vector of 1 million simple elements? Intuition says about a minute, but it's milliseconds. How big a working set before cache thrashing? My intuition is attuned to single-digit kilobytes, but it's multi-digit megabytes now.

Why would intuition say that sorting 4MB of data is slow?

Assuming the worst case with 4 billion elements:

log_2(4 billion) = 32

So 128 billion operations or 500GB of memory read and writes if everything is touched only once. Anything below that is peanuts.

You need unimaginably large numbers to make sorting slow. How am I going to find that much data?

Is there a reason why AMD hasn’t gone past dual-socket setup to allow for this type of extreme scaling?

I assume it’s simply due to market size being so small for a server so large?

Also Infinity Fabric may have poor performance beyond two sockets.

There's a diagram at https://www.storagereview.com/news/3rd-gen-intel-xeon-scalab... which does a good job showing the links between individual CPUs. There's a performance penalty to access data in memory attached to other CPUs, which reduces use cases.

> ” There's a performance penalty to access data in memory attached to other CPUs, which reduces use cases.”

Isn’t that NUMA.

And even AMD has this issue with single socket chips because their chiplet design is NUMA.

AMD had higher socket counts in earlier HW (Opterons).

Do these still come packaged with Chinese state-sanctioned malware that auto-installs itself from the UEFI/BIOS on every boot?

How much does something like this cost?

A quarter million maybe.

I speced out a 4 sockets sapphire rapids dell server with 16tb ram, it came out to over 1.2M, so imagine with 32tb... But really no one should be paying retail for these.

It's 480 cores. Typo?

Yup, just edited, thanks.

32TB ought to be enough for anybody.

!RemindMe 10 years

Well-played!

I am old enough to remember the original quote that I am almost certain you are referencing.

https://www.computerworld.com/article/2534312/the--640k--quo...

Who is still buying tower servers? Seems like a time machine back to the 90s but there must be a market....somewhere?

Small businesses in particular.

I remember back in the day working in companies of 15-40 people. There'd be ONE server. That ran everything you needed -- auth, DNS, file sharing, email server, application servers, web servers for internal apps, database server for internal apps. Usually sourced from a local MSP - or at least you had a support contract with them.

You didn't need a rack. You also had something that wasn't designed for a rack, so it could be in a secure back room with the switches, a UPS, a router, maybe some external storage... where it'd just sit quietly. Just has some big slow fans because that's all it needs.

Trust me, still a thing in small businesses, small retail stores, restaurants, etc.

I do a lot of work with AI and I love towers.

Basically, they're much much quieter than a 1U or a 2U server. I once tried using a HP DL320 in my office and the noise from the fans was unbearable.

Apparently you can replace 1U server fans with much quieter 40 mm Noctuas.

Reply on news.ycombinator.com