Hacker News Reader

I'm running a server motherboard (Supermicro H12DSi-N6), and while the processing power is wonderful, the tradeoffs were a bit surprising:

* I can't disable the BMC video card, which means that when I'm running Windows, there's a 800x600 monitor that will never go away and windows occasionally decide to open on it. This is solvable by opening the BMC on the machine itself and using the IKVM webapp to drag it onto my main screen; this takes a very careful flick of the mouse. * There's no built-in audio; in order to listen to Spotify, I use a USB audio card. * The board only has 6 USB ports, only 2 of which are USB-3. Thus, I get to use a lot of hubs. * It takes a long time to boot past the firmware; generally 3-4 minutes to warm boot and 6-8 cold. * While it can be used as a single-socket board (which I do, because I don't have the funds for a second CPU just yet), I lose a bunch of PCIe slots doing so; half the PCIe lanes for the first CPU are direct wired to the second socket for the inter-CPU link.

Bridge jumper JPVGA1 directly on the motherboard and it will disable the onboard VGA. Switch BMC priority from onboard (default) to offboard, I don't think this is necessary if you bridge JPVGA1.

Get yourself a nice Highpoint USB 3.0 card with quad controllers on each port. 20Gbps of bandwidth in aggregate.

I believe there is a jumper on the motherboard that will bridge the missing PCIe slots in single CPU configuration.

If the 800x600 is on the right and a window opens in it, you can alt-tab to it or click it on the taskbar, then just hit win+leftarrow until it appears on the correct monitor. Same for win+rightarrow if other way around. Should not need to IKVM

On all Windows operating systems I have used, this workaround (winkey+left/right) unfortunately does not work for moving modal pop-ups.

For example, when starting KeePass by opening a .kdbx password safe file, KeePass will first open the master password input modal pop-up.

If that happens on a turned-off screen (IMHO it opens on the screen on which the KeePass main window was placed before the process exited on the previous run), you'll be lost.

Very frustrating, e.g. for multi-screen home theater setups.

Anyone please share your solutions to this problem.

Moving a window: Alt-Space, M, arrow keys

If you Alt-Space, M, and press one of the arrow keys, the windows will 'stick' to your mouse cursor so you can just wiggle your mouse after they key combo to bring it to you.

I was inspired by nostalgia to do the research via archive.org: This shortcut goes back to Windows 2.0, almost 35 years ago. I remembered it from the 3.1 days.

The old CUA, Windows 3.0 corner square has an oblong rectangle that represents the space bar (at least in my mind). Unless the window is a child window, in which case the shortcut is Alt-Hyphen[1], and I haven't used that shortcut in many years. The fashion for tabs has probably made that useless.

[1] http://www.robelle.com/smugbook/win3x.html

Win + Shift + arrow works better with modals because it doesn't try and resize/pin the window but just blips it to the next screen over preserving size and location. I use it a lot when I move from my laptop to a multi-mon setup to sort windows to different screens quickly.

In case helpful, select the app on the task bar and use Shift+Win+Left/Right arrow key to move it between monitors without needing to do the IKVM dance

If that's the only other monitor, you can set your actual display as primary then select primary display only in Win+P menu

Can you disable the video driver in the system control panel if you don't want Windows to use it?

Other stuff seems normal for a server board though.

| * It takes a long time to boot past the firmware; generally 3-4 minutes to warm boot and 6-8 cold.

^^ why is that? is it just the firmware being buggy, or is there any legit reasons why they need to self-check _for minutes_?

Server hardware operates remote and headless, so it performs much more exhaustive memory checks and will try to proactively report failures. Good datacenters will catch these and open a ticket to replace failing hardware.

Beyond that, lots of servers will then enumerate all of the network interfaces and try to initiate a net boot, and only then fall back to the hard drive. Some tweaking in the bios can usually disable these features and gain a bunch of time, but again in a headless remote context this default behavior is incredibly useful for recovery purposes. We like to avoid actually running out with a physical crash cart if we can help it :)

Damn, respect, your mainboard alone is probably worth more than my entire workstation.

You could likely use autohotkey and winmove to place the modal windows where you want, I'm not at my desk now, but I'll update this post with the script I use (or a reply to it).

> One related issue is the death of integrated PCIe switches on motherboards. High-end boards used to have PCIe hubs on them so you could plugin in a lot of cards and still get all the bandwidth out of a 16-lane CPU.

PCIe switches weren't common or necessary for most consumer applications. Workstation and server CPUs have plenty of lanes on their own. PCIe switches occupied a relatively narrow use case in the middle where people wanted to use consumer CPUs but attach a lot of add-in cards and share bandwidth across them.

Even that use case has been eroded a bit with PCIe splitting options, where an x16 lane can be split into x8/x8 or even x4/x4/x4/x4 depending on the motherboard. An x4 PCIe 4.0 lane is as fast as an old x16 PCIe 2.0 lane, so that's a significant amount of bandwidth still. PCIe 5.0 takes this even further, where an x16 PCIe 5.0 slot has over 60GB/sec of bandwidth, which is a lot even when divided into x4 or x2 channels.

AMD especially made platforms with a lot of PCIe lanes relatively affordable. An AMD workstation motherboard and CPU with a lot of PCIe lanes might not actually be that much more expensive than a hypothetical consumer CPU with a PCIe switch. The other issue is that with a PCIe switch is that you're still limited to upstream bandwidth, so if you need x16 bandwidth running to multiple cards at the same time, a PCIe switch doesn't actually help you.

It's still a problem if you want to run previous gen hardware. For instance, if I have a Gen4 motherboard and pass through a second GPU to a VM - so the second GPU at Gen3 x8 plus the host GPU at Gen4 x8 - I'm only using a quarter instead of half the throughput the hardware actually supports, because Gen3 is half the speed of Gen4 per lane.

Or even think about Gen5, doubling Gen4 speeds: With a PCIe switch, I could get essentially 3/4 performance out of a Gen5 host GPU, and full performance out of a Gen3 second, instead of 1/2 and 1/4 respectively.

Do GPUs even saturate PCIe3 x16 or x8 connections, let alone PCIe4 or 5, that we need to be worried about bottlenecks?

The only thing I'm aware of that will actually saturate a PCIe connection is NVME storage.

For gaming, not even close. I've often had a GPU monitor running, charted bus utilization...and it's barely a few percent no matter what game I'm running, except for a very brief spike during texture and model load-in at the start of a round/level.

Specifically for VM's, something like Looking Glass (shared framebuffer so you don't need a dedicated monitor) needs the extra bandwidth to get the video signal through without tanking performance, especially at higher resolutions and refresh rates.

Aside from that, I agree it's not too big an issue for 95% of computer users, but it's something to keep in mind.

[1]https://looking-glass.io/

To quantify this: a 4k 60Hz uncompressed video stream needs approximately one lane of PCIe gen4 (though only in one direction, while PCIe has the bandwidth in both directions). So any realistically obtainable resolution and refresh rate will not be a problem for sharing the framebuffer on a GPU with PCIe gen4 x16, but lower-end GPUs often have just x8 and there are even some mobile GPUs with x4 being packaged onto desktop graphics cards now.

I've had significant frame rate drops with PCIe3 x8 compared to PCIe3 x16 (Easy 20 - 40% drop, depending on the game)

If I remember correctly, running a 3080 10GB in PCIe 4.0 yields >5% performance average in gaming benchmarks compared to running it in PCIe 3.0.

You could probably argue that you should just settle and not take the 5%, but when you are buying in at such a price point, you are likely concerned with 5% differences. There are also higher end SKUs such as the 3090Ti where that performance gap is likely larger.

This is precisely why I am not going to buy into this year's Ryzen motherboards, how am I supposed to attach all of my PCIe 2.0 LSI HBA as well as my 2 GPUs, and NVMe? My options are buying 2 machines (one for storage, one for compute), sacrificing speed somewhere, or buying into either Threadripper Pro OEM or EPYC ($$$).

The bandwidth is there by an order of magnitude, but not the lanes to actually use it.

It's pain that even PCIe splitting isn't adopted well on recent Z690/X670 mobo. So they just provide us a PCIe 5.0 x16 slot that is unnecessary great but practically nothing better than 3.0 x16 for gaming (and most workload) now . I'd like to have PCIe 4.0 x8/x8 rather than that.

PCIe bifurcation is a function of what your CPU and/or PCH can do. There are hard limits in the silicon as to how you can divvy up your lanes.

Splitting CPU lane to x8/x8 is supported at least. Just mobo manufacturers don't want to implement it as a dedicated slot. It seems that Alder Lake no longer support x8/x4/x4 sadly.

> It seems that Alder Lake no longer support x8/x4/x4 sadly.

I would be very surprised if the silicon doesn't still support 8+4+4 operation. I suspect the lack of available motherboards with this config comes down to the expense of getting a PCIe gen5 signal to reach the third slot. It may require both the extra mux and a redriver or retimer.

I found this from quick Googling https://forum-en.msi.com/index.php?threads/does-pro-z690-a-a...

Perhaps why they dropped the support is because the reason you wrote? Still I wonder is it possible to have Gen5 x8 and Gen3 x4/x4.

> It seems that Alder Lake no longer support x8/x4/x4 sadly.

Sort of. support is "Up to 1x16+4, 2x8+4" [1] so could be used as x8/x4/x4, the remaining 4 lanes are unused in the x8, as there are 20 PCIe lanes on the CPU.

[1] https://ark.intel.com/content/www/us/en/ark/products/228441/...

Ah yes, but everyone (and every manufacturer) want to use +x4 port for M.2 SSD so not usable for other PCIe devices.

>PCIe switches weren't common or necessary for most consumer applications. Workstation and server CPUs have plenty of lanes on their own. PCIe switches occupied a relatively narrow use case in the middle where people wanted to use consumer CPUs but attach a lot of add-in cards and share bandwidth across them.

PCIe switches still have a place in storage servers. I can build a backplane with a PCIe switch so I can get a ton of ports/slots for HDDs or SSDs and not have to buy a much more expensive CPU with the IO to handle everything. The trade off is that you are over subscribed and all disks will never reach full bandwidth simultaneously.

AMD definitely has a recent high end desktop/workstation platform with its ThreadRipper Pro WX CPUs (EPYC server CPUs) that have 128 PCIe 4.0 lanes and 8 memory channels. And enough PCIe slots to actually use this throughput with the right kind of hardware, like a bunnch of PCIe4.0 NVMe SSDs [1].

Yep, the platform is 2 years old, but supports both the Zen2 and current Zen3 CPUs (but not PCIe5.0 & DDR5 as Zen 4 requires a different CPU socket and is only announced right now, not available from AMD until the end this month).

However, 8 x DDR4 channels and 128 x PCIe4.0 lanes still give you better total bandwidth than 2 x DDR5 and 24 x PCIe5.0 like the few currently available DDR5/PCIe5 platforms support.

[1] https://tanelpoder.com/posts/11m-iops-with-10-ssds-on-amd-th...

The unfortunate part is that Threadripper nowadays is OEM only, thus it is very expensive.

You can just switch to EPYC with hardly any difference in price point.

The Threadripper Pro 5000-series is available for retail purchases [1].

When 3000-series was launched, it was first OEM only (Lenovo) but later became available in retail too.

[1] https://www.amazon.com/AMD-ThreadripperTM-PRO-5975WX-64-Thre...

Oh nice, I must have missed this. Motherboard prices are leaving something to be desired (lowest price is $900), but still, this is good news.

Edit: On second glance, the markup compared to last-gen Threadripper is gnarly. 24 core SKU MSRP is $2.4k, compared to the 3000 series SKU which MSRP for $1.4k.

At that point just go EPYC.

And other creature comforts like decibels, I suspect.

AMD Threadripper got an update this year, and a new release should come early next year. 128 PCIe lanes.

Can confirm, much of my high-compute architecture design work centered around heavy analysis and comparison of existing and promised backplanes. Especially when you throw images in the mix and need gpus, but you need the whole stack to be HC/HT or you hit bottlenecks later.

I'm raise you no working video output on my consumer motherboard! cries in GPU-less AMD CPU

How do all these lanes get into the CPU? There are only so many pins on a chip. Are they time-shared? If so why do you need n physical slots and not just a switched network that lets you daisy chain as many as you can switch? (I’m an architecture lay-person.)

Threadripper comes in a seriously big package. 4094 pins, compared to 1718 for AM5 (the new ryzen socket).

PCIe already is a packet-switched network, but the big and fast switches are very expensive.