High-speed 10Gbps full-mesh network based on USB4 for just $47.98
fangpenlin.com
fangpenlin.com
And What happen if you wire both ports together on the same PC?.. Do you get a broadcast (thunder) storm?
Thunderbolt was also expensive, which is why adoption was limited, but it's becoming more maisntream since Intel and Apple have been pushing it in the last years, adn piggibacking over USB-C makes it an easy sell comapred to requireing a separate connector like firewire.
Still, thunderbolt peripherals are way more expensive than USB ones, so like Firewire before, use is still more in the enthusianst/professional space.
Compared to Gigabit Ethernet back in that time period? Firewire was a huge bargain.
Completely loading a 5 Gig iPod with music over that first version of Firewire still took a few minutes.
Yes and? At what price points? What was the adoption rate? How many mainstream PCs and peripherals worldwide had it?
Wherever you went, whoever you met, you were way more likely to find a USB or ethernet port to hook up for a fast transfer rather than Firewire.
At least in my country at the time, maybe you lived in Cupertino/Palo Alto where evryone had iMacs and firewire.
Just like VHS over Betamax, USB won because it was cheaper and more convenient despite technically inferior to firewire and consumer tewch at the time was a race to the bottom in terms of price.
>Completely loading a 5 Gig iPod with music over that first version of Firewire still took a few minutes.
Only the first gen iPod had firewire before switching to USB, and even then, what was the point of Firewire 400 on it when the tiny and slow mechanical HDD on it was the real bottleneck.
There was no way the iPod would have been remotely as successful had it stayed on firewire. Apple didn't have the market sahre back then to enforce their own less popular standard. Only when it switched to USB and supporting PCs did the iPod really take off.
At the time, USB was still limited to 12 megabits per second and transferring that same 5 Gigs of MP3 files would have taken over an hour. The firewire iPod did it in a couple of minutes.
USB was cheaper, but dog slow.
Gigabit Ethernet was faster but WAY more expensive.
Again, only the first gen iPod was firewire exclusive and it was not yet a maisntream product since it was still Mac only, so avergae consumer demand at home computers for Firewire was lackluster and the iPod didn't change that.
Firewire was niche or non existent in the home PC space and it died completley with the launch of USB 2.0 remaining alive only in the pro-sumer space.
>Gigabit Ethernet was faster but WAY more expensive.
Please show me where I mentioned Gigabit ethernet as an argument. I said 100 Ethernet which was dirt cheap and almost every PC and Mac had it, as opposed to Firewire, so if you needed a fast cross platform transfer it was your best bet at the time in terms of cost and mass availability over firewire before USB 2.0 and gigabit hit the market.
Which completely ignores the speed and the costs of the various data transfer standards as they existed at the time.
The cheap 1.2 megabit USB standard that existed at the time couldn't transfer 5 Gigs of MP3 files in less than an hour.
The cheaper 10 megabit version of Ethernet they sold at the time also would need more than a hour to transfer enough MP3 files to fill an iPod and wouldn't have been cheaper than a Firewire port.
Ethernet with faster speeds than 400 megabit Firewire existed, but was MUCH more expensive.
Speed AND cost both matter.
> I said 100 Ethernet which was dirt cheap
Back then? It wasn't.
FireWire at 800Mbps beat Gigabit Ethernet in terms of latency for a rather hard real-time system.
The low cost is price. Asus for example with their ASUS ThunderboltEX 4 allows you to have TB4 via PCIe card.
The neat thing about USB4 was same as PATA and later SATA: widely and relatively cheap available in consumer hardware. SCSI and FireWire were technically superior but were neither cheap nor widely available.
Oh and I don't know about SCSI but FireWire was actually a security risk.
Luckily there are good alternatives. I landed on a solution using a Belkin[0] DisplayPort and USB to Thunderbolt-cable. I just get USB2.0 speeds, but it's enough for my needs. I'm also able to extend it using an active DisplayPort 1.4 extender, for a total of 10 meters cable.
[0] https://www.belkin.com/support-article/?articleNum=316883
10 Gbps doesn't come close.
Of course this example is still interesting and cool.
It seems 25Gb dual port Mellanox CX-4 cards can be found on eBay for about $50. The cables will be a bit pricey. If not doing back to back, the switch will probably be very pricey.
Each lane of PCIe Gen 3 can deliver 985 MB/s [2], meaning the typical drive that uses 4 lanes would max out at 3.9 GB/s. Surely there is some PCIe/NVMe overhead, but 3.5 GB/s is achievable if the drive is fast enough. There are many examples of Gen 4 drives that deliver over 7 GB/s.
Supposing NVMe-oF is used, the MVMe protocol overhead over Ethernet and PCIe will be similar.
1. https://enterprise-support.nvidia.com/s/article/roce-v2-cons...
https://www.ebay.com/itm/273064154224
And yep, they do apparently work ok if you're running Linux. :)
https://www.youtube.com/watch?v=dOIXtsjJMYE
Haven't seen info about how much noise they generate though, so not sure if suitable homelab material. :/
Cheap though... it's a small fraction of the price for a new one.
Are there better options around?
Looking at Ebay just now, I'm seeing some Mellanox IB switches around the same price point. Those things are probably super noisy though, and IB means more mucking around (needing an ethernet gateway for my use case).
A literal loopback interface? Two of them, most likely.
Do you know if this is the case for all thunderbolt generations (speed differences aside)? Does it apply to thunderbolt using mini DisplayPort too or only over USB PHY?
https://www.gigabyte.com/Press/News/1140 this doesn't mention it and the Intel whitepaper specifically requires TB2 so I would guess TB2 was it.
It should've been PCIe 2.0 x1 and then 3.0 x1. There was absolutely no reason not to do it: PCIe 2.0 came out in January 2007, USB 3.0 came out in November 2008. PCIe 3.0 followed in November 2010 and 10gbps over the USB C connector didn't appear until August 2014.
What USB4 version 2.0 can only do with a complex tunneling architecture we could get "straight": PCIe 5.0 x1 can do 32gbps which closely matches the 40gbps lane speed defined in USB4 version 2.0 (which again came out years after PCIe 5.0 mind you). It would require two lanes, one for RX one for TX and the other two lanes could carry UHBR20 data for display, for a total of 40gbps. This very closely resembles the 80gbps bus speed of USB4 version 2.0 but the architecture is vastly simpler.
We wouldn't have needed dubious quality separate USB-to-SATA then USB-to-Ethernet etc adapters. External 10GbE would be ubiqutious instead of barely existing and expensive. Similarly, eGPUs would not need to be a niche and DisplayLink simply wouldn't exist because it wouldn't need to exist and the world would be a better place for it. You could just run a very low wattage very simple but real GPU instead. Say, the SM750 is like 2W.
https://www.apple.com/shop/product/MMEL2AM/A/thunderbolt-3-u...
Nice one Apple. I think I'll buy this, a cable and a cheap tb3 dock.
I have two identical machines that I need to do this with... lemme test it and see.
...
ARGs usb-3... so nopes. (How FN lame is that)
In terms of scaling this to multiple hosts as the author does, I've read that it is possible to daisy chain, or even use a hub, but it doesn't strike me as the most reliable way to build a network. For an ad hoc connection, though (like null modem cables of yore), it's a great option.
As far as points of failure, there's no additional hub/switch in between the devices, so you have a Thunderbolt controller on each device, two cables, and two ports. If a cable goes bad, so long as there isn't a silent/awkward failure mode, all three nodes can still talk to eachother, at degraded speed. If a switch goes bad, the whole network is down, unless you start talking about redundant switch topologies.
To your point though, there does seem to be plenty of shenanigans with performance, especially between devices with different Thunderbolt controllers, that may make this less ideal. But IMO, that's more of a question of do you want to with a more battle tested topology, or are you okay with a less battle tested, but still highly performant and "simple" (we won't go into how bonkers the USB/Thunderbolt spec is) topology?
OK but what then? I've had ethernet ports on my computers since I can remember, and that hasn't magically allowed me to transfer data back and forth just by plugging a patch cable into both machines. What software is at work here?
When you connect a patch cable directly (no crossover cable needed in the 21st century), you'll likely find that each system has a self-assigned IP in the 169.254 network.
With the caveat being that's only if you're not up for adding PCIe cards.
If you are ok with adding some PCIe cards, then you can transfer things a lot faster than 1GB/s. :)
They're widely used (with many different types) in IT data centres, home labs, and probably other places too.
Heaps of them are on Ebay. As a random search just now for "Mellanox 25GbE" on US Ebay:
* https://www.ebay.com/itm/134435757546
* https://www.ebay.com/itm/355348422765
(there are hundreds of individual results)
Searching for "Mellanox 50GbE":
* https://www.ebay.com/itm/225021493021
* https://www.ebay.com/itm/233915360659
(less results)
There are older generation ones too, doing 40GbE:
* https://www.ebay.com/itm/305046322527
* https://www.ebay.com/itm/166350081025
(hundreds of results again)
With those older generation cards, some care is needed depending upon the OS being run. If you're using Linux you should be fine.
If you're running some other OS though (eg ESXi) then they might have dropped out of the "supported list" for the OS and not have their drivers included.
---
Oh, if you have even a hobby grade CNC machine around, you can get a fairly professional level result:
https://forums.servethehome.com/index.php?threads/3d-printab...
One idea for why they were only able to reach 11Gbps is having only one Thunderbolt/USB4 controller[2], meaning the two USB4 ports split the 40Gbps PCIe lane. Throw in a full-duplex connection and you get 10Gbps in one direction.
[1] https://youtu.be/GqCwLjhb4YY?t=81 [2] Just a theory but seems like a sane assumption.
The comparison with the Dell r630 power numbers got me interested since I just purchased a Dell r430 to host my site so I decided to benchmark mine.
Specs:
* 2x Xeon E5-2680 v3 (same CPU)
* 64GB RAM
* 2x power supply (can't remember if it's 500W or 750W each and too lazy to look)
* 1 SSD & 1 7300 RPM HDD
* Ubuntu server 22.04.3 LTS
If my calculations are correct that's 72 kW per day or $11.05 per month at idle:
0.1 kW * 24 hours * 30 days = 72 kWh
72 kWh * 15.34 cents/kWh = $11.05
0.22 kW * 24 hours * 30 days = 158.4 kWh
158.4 kWh * 15.34 cents/kWh = $24.3
Lack of understanding. Even comparing 65W to 1000W should had ring some bells, but.
> but these numbers are probably more realistic
Almost, depends on the load (hardware) and load (software), as someone who manages a fleet of 720/730/630, a standby server eats around 150W and under the load up to 300-350W, depending on the package.
> Using a Kill-A-Watt meter
You can use built-in measuring in iDRAC.
I didn't know iDRAC could measure real time power usage. Pretty amazing.
The machines each have 192GB of ram, so I thought I'd set them up as LLM inference machines. I figured that with that much ram, I could load just about any model.
Then I discovered how slow the CPUs on these older machines is. It was so utterly slow. I have a machine I bought from Costco a few years ago that was under $1k and came with a RTX 3060 with 12GB of GPU ram. That machine can run around 20+/tokens per second on 13B models (I actually don't know - I stream the text, and cap it at 9 tokens per second so I can actually read it).
The R420? Its tokens per second were in the 0.005 to 0.01 range.
So, yeah, not a good CPU for that sort of task. For other stuff, sure. I thought I'd setup a small file server with one instead, but the fans are so jet engine loud that it's intolerable to have in any part of the house, even when managing fan speeds with software.
According to iDRAC, my R630's are drawing almost exactly 100w/each. Each with about 75 pods (k8s nodes).
Does anyone know if ~modern HP iLO can show power usage too?
Looking through my Gen8 HP Microservers just now, I'm not seeing power usage info. :(
"Power Information" is just showing the state of the power supplies (aka "OK", "Good, In Use") and not much else.
It's just your MicroServers are gutted so they wouldn't compete with enterprise gear and ML line (probably).
Try:
SNMP
IPMI
HPE Agents (Linux, Windows)
REST/RIBCLI interface for iLO (https://support.hpe.com/connect/s/softwaredetails?language=e...)
It certanly has some reading on the power but it's just don't display it in the interface.
At least https://www.storagereview.com/review/hp-proliant-microserver... says it has Power Meter menu entry?
Also check if you really running the updated firmware for the iLO.
The Power Manager is unavailable for this configuration.
Haven't bothered with setting up SNMP nor IPMI on the Microservers though, as it seemed like a bunch of effort for no real benefit (for a homelab). The HPE agents thing is an idea. I'll have to take a look at what that needs, can hook into, etc. :)
These Microservers are definitely running the latest firmware, and the latest iLO firmware too (released last year).
A "real" ring is formed when you have more than 3 nodes and so some destinations require more than one hop. A mesh implies a network formed by somewhat arbitrary point-to-point connections with multiple possible routes between two points.
What is usually described as a mesh network (technically: "partially connected mesh") is a network where every node has a route to every other node, and that every node may have a direct connection to any number of nodes, but where there is no guarantee that any two nodes have a direct connection and may need to route over other nodes. The ability to handle this sort of completely arbitrary network topology is the core to mesh networks.
See https://en.wikipedia.org/wiki/Mesh_networking. There's also the old WiFi mesh spec (https://en.wikipedia.org/wiki/IEEE_802.11s, what the OLPC used), not to mention Zigbee and Bluetooth Mesh.
If routing was enabled this would definitely be a mesh network. It would be a full mesh network where every node is directly connected to each other and a secondary path through another node is available should the direct path fail.
This is not a ring because it is not configured as such: Rather, each network card has exactly one peer, and there is no knowledge of other peers, making it no different from just a direct connection between two computers.
If you had redundant paths, it would be a ring yes. A 3-node routable ring is in theory a special-case of a fully-connected mesh, but you do not call ring networks mesh networks. Mesh is usually reserved for when you truly mean to support arbitrary, disorganized network topologies
And a 3 node ring is absolutely still a ring network.
Yes, that is a classical partially connected mesh because it functions without a direct connection between all nodes. That you started out fully connected does not matter.
If the network relies on direct connection between source and destination, it is not a mesh.
> And a 3 node ring is absolutely still a ring network.
Only if it can route when you break the loop, and only if you extend it by inserting more nodes into the ring.
The example in this article does not route, which is why it is absolutely not a ring - If you disconnect server A and B, they cannot talk even if they both connect to C. There are just 3 entirely independent point to point networks.
My favourite was bootstrapping using 'COPY /B COM1: C:LL3.EXE' to get the Laplink executable to the target machine over Serial, when you didn't have a spare floppy.
Where Laplink isn't really a network, just a file transfer thing that requires Laplink to be running on both PCs, PLIP is a network driver that lets you do all the usual network things over the connection.
And since PCs can have up to 3 parallel ports before things start getting stupid, it's pretty straightforward to have a row of machines with PLIP links going both ways, bridging or routing the interfaces. Or, do PLIP-SLIP-PLIP-SLIP without adding any ports, and you could have a functional-but-brittle-and-slow network for pennies.
I was running that in the nineties. My main desktop, running Linux, and an ultra old, ultra crappy laptop running Linux too. They'd be connected using PLIP and the desktop, more powerful, was running its own X server but also applications for that were running on the laptop's X server.
So my brother and I could both be using Netscape to surf the net (we'd call it that back then) at the same time, over the 33.6 modem connection.
It was really easy to run PLIP and was saving me the trouble to try to get network card running under Linux on my desktop and most importantly saving me the trouble to try to get the PCMCIA crap to work on my laptop.
Fun times...
P.S: and, yup, back then laptops had a full parallel port!
Nope but I would loved that back in the days: we created a LAN between our house and the (attached) neighbors' house (so we could play Warcraft II against each other) but... We couldn't create a LAN with the neighbor across the street!
Once around 2005-ish, I scored an 802.11b client-bridge real cheap because .11g stuff had been out a while. Velcroed it to the lid of my Zenith Supersport, and made a ten-inch ethernet cable to connect it to the PE3. An unholy abomination allowed both units to tap power from the keyboard port; the less said about that, the better.
What felt like thirty hours of hair-pulling later, I had a 720k DOS boot disk with packet drivers and a telnet client, and I could MUD from my lap, wirelessly. Ahh, the sweet smell of useless success.
Then like an idiot, I sent all that stuff to the recycler around 2008.
Edit: Yes, I'm aware that box says RS-232, but with gender-changers, the dip-switches and some cable-bridges, you could abuse it for 'parallel'.
IP-over-SCSI was great, you could throw 8 PCs on one SCSI chain. Put two SCSI controllers in each machine, do rows-and-columns, and you could have 64 hosts a maximum of 2 hops from each other, at U320 speeds, in the 1990s.
https://www.linuxjournal.com/article/2344
Imagine a beowulf cluster of hot grits! Er, sorry...
I've never heard of it, but given that it's possible, could you do theoretically do IP-over-SATA?
https://lore.kernel.org/linux-ide/4D6A5B72.6040600@teksavvy....
Company specific TB driver/firmware updates can be found here https://www.thunderbolttechnology.net/updates
I cannot find any specifics regarding the ports on this 790 Pro one, but I'm certain that's ultimately the issue: the bottleneck is on that second (the one on the right) USB4 port.
average residential electricity rate is 15.34 cents per kWh
The linked page cites data from over a decade ago (2012).
Within the state, there's huge variation. The average is around 25 cents. E.g. if you live in Santa Clara, you pay 16.6 cents per kilowatt hour to Silicon Valley Power, while all surrounding cities pay 45 cents-ish.
Their data might be a bit outdated but the December 2023 average is $0.168/kwh according to [1].
[1] https://www.bls.gov/regions/midwest/data/averageenergyprices...
The page you linked shows four California cities, each with Dec 2023 rates over $0.27/kWh.
While their solution is more livable for them, the hardware is vastly inferior for actually hosting serious services on, and they don't seem to understand that because they're software guys who're getting away with it.
Electricity in free markets is more like 5-8 cents per kWh.
The interface/bridge MTUs might want increasing to a larger value. I notice a pretty big difference when connecting Macs together using Thunderbolt with an MTU of 9000 vs 1500.
Hopefully there's wins available & this isn't some silly unadvertised hardware gotcha. Higher mtu, trying some parallelism are two good suggestions I heard.
One other thing I'd note: it's only been very recently that Linux has learned how to let the USB and DisplayPort negotiate/allocate bandwidth. It was evenly split between the two until March. Linux 6.3 Adds Thunderbolt/USB4 DisplayPort Bandwidth Allocation Mode. Linux 6.3 Adds Thunderbolt/USB4 DisplayPort Bandwidth Allocation Mode
It's unclear how automatic this bandwidth management is under what systems; users might need to use the new thunderbolt-utils from July to adjust it manually. Intel Rolls Out thunderbolt-utils To Manage USB4/Thunderbolt Devices On Linux. https://www.phoronix.com/news/Intel-Linux-thunderbolt-utils
I really want to hope much better is already possible. I don't own any USB4 systems though! How awesome they are. I hope we see some cool all-in-one's with multiple USB4 in; upcoming Minisforum V3 tablet for example has at least one usb4 that can do DisplayPort In, if I understand, for example, and that capability feels like it should just be coming for free now on PC's USB4 ports!!!
Alright, a server will be more power hungry than e.g. your desktop, but... This specific Dell R630 has 2x 750W PSUs in it. That 750W is the maximum rating of one power supply, and there's 2 of them for redundancy - not for increased power intake. That server will run at 750W maximum - but that is the absolute, absolute maximum power it should draw. It's when you have all the rails loaded to the limit and running the server to the ground.
A more realistic scenario would be e.g. 100W or so on average.
The worse problem if running this server at home would be the terrible small high-RPM fans they have in 1RU servers. The loud high-pitched whine of them will drive you nuts. A better idea would be to get either a lower-power 1RU pizza box, or something larger that can take larger fans - replacing the fans with something quieter and adjusting the fan controller to spin at lower RPMs.
>1000W per hour
This is just wrong.
I don't think it uses much power when idle either. I think that rack servers being expensive is a myth.
Adjusting the RPMs in software helped a little, but even at the lowest speeds, it was a hovercraft. It had to go.
Ahem:
System Headroom
Statistic Reading
Instantaneous 1528 W | 5215 BTU/hr
Peak 1346 W | 4594 BTU/hr
Hot spare is a power supply feature that configures redundant Power Supply Units (PSUs) to turn off depending on the server load.
*This allows the remaining PSUs to operate at a higher load and efficiency.*
This requires PSUs that support this feature, so that it quickly powers ON when needed.
Redundancy Policy:
Not Redundant — In this mode, failure of a single PSU can power off the system.
Input Power Redundant — In this mode, the system is functional in the event of failure of a PSU input circuit,
provided the PSUs are connected to different input circuits. This is also called AC redundancy.
But yes, these guys idle at 150W at have around 300W under load with dual CPUs and a lot of RAM.
For throughput try sticking "--parallel 4" or shorthand "-P 4" on the iPerf3 command and see if total throughput changes.
I'm bad at low-level networking, but could it be a routing issue? Effectively every machine is also a router, so there might be some wasting going on.
Why did they set a limit so low is not known, but the supposition made by another poster that this is a market segmentation feature may be right, because such policies have always been typical for Intel.
At 10gigabit/second, with 1518 byte packets, it's 823451.9104 packets per second.
So in a single stream, you have to process each packet within 1.21microseconds to keep up
At 20gbps, you have 600nanoseconds per packet.
There are also almost certainly timing/synchronization issues between different stacks like this. It's horribly inefficient.
Network cards achieve >10gbps in part by offloading a lot of stuff.
Even if the CPU can handle the load, just going through different stacks like they are may add enough latency to throw single stream throughput off.
The posited reason of "not compete with network cards" is beyond stupid. It can't because you can only do a few meters this way, max.
That's not interesting at all. 25gbps network cards are cheap and for a few meters, a $10-15 25gbps DAC will suffice
For more than that, 25gbps transceivers are 25 bucks a pop for 25gbps-SR, which will do 100meters.
With none of the problems of trying to use longer thunderbolt cables, too.
Intel's 25gbps SKUs are not where they make their money anyway.
In the bay area it's 2.5x - 3x the rate.
I stopped using my rack server for that reason. I was dumb and didn't consider the electricity costs when buying mine at the time years ago. It sits around collecting dust now.
The author's alternative idea is really good considering 8w idle and 80w peak and I doubt it hits peak for what the author wants to do.
(What I really wanted was the low latency, but the bandwidth is handy to have sometimes)
The article also seems to make the assumption that a server would be pulling 1000W all the time, 24/7, which is rarely the case (of course, I can't comment on what their workload might be, but it would be quite unlikely)
Still, I like the direct networking being done here. But saying "you can build a 10Gbps mesh for $50" when you have 3x $750+ machines seems a bit disingenuous. It is not unreasonable to get 10Gb SFP+ NICs on ebay for ~$50 a pop ($150 for 3)
1x Juniper EX3300-24p - $75
7x SFP+ optics - $7/each
3x Intel X520-DA2 NIC - $20/each
4x 3 meter OM3 LC-LC fiber - $6/each
1x 30 meter OM3 LC-LC fiber - $24
-----
Total: $232
The EX3300-24p has 24x 1gb copper ports with PoE+ on them, and 4x SFP+ ports. If you need more SFP+ ports you'll want to find a different switch - but for a small multi-use home network the EX3300-24p nicely matched my requirements.
CRS309-1G-8S+IN, Suggested price $269.00
>> Desktop switch with one Gigabit Ethernet port and eight SFP+ 10Gbps ports
Personally I have a love/hate relationship with MT, with a little love and a lot of hate, but at their price range they are unbeatable and works 99% of time.
https://www.servethehome.com/mikrotik-crs309-1g-8sin-review-...
Despite being great in most ways, AMD desktop systems in general rarely have thunderbolt (unlike Intel) - a few niche motherboards have it. You also can't just add it with a PCIe card, it requires special support from the motherboard.
Hopefully this might change with USB4 not sure.. be interesting to see what the lower end motherboards end up shipping support for.
I simply set up static IPs on each one and also setup internet sharing from one to the other. Surprisingly barrier, internet sharing and even charging always works with this arrangement. I only have to connect 1 macbook to wifi and power.
The only caveat is that I am running a 10Gbit ethernet network at home and it wasn't that costly to setup. A 10GigE switch costs around $500 CAD right now and that is all you need.
Surely your machines all need 10GbE NICs as well? Admittedly my hardware isn't the newest (2.5GbE at most), but from a quick search 10Gig PCIe cards are around $150 each. Meanwhile 10-20 Gbps USB3.0 is reasonably common already. Though, using Ethernet still has many other advantages over running USB C cables everywhere.
With a multigig capable SFP+ module it can handle 2.5G/5G copper as well.
That being said, neat concept IMO.
Any particular advantage/disadvantage to doing it this way?
The main disadvantage inherent in the physical topology is that you are always going to do switching/routing decisions on the CPU. But as long as you use that as a virtualization platform or run k8s on that you are going to do that anyway and the additional overhead is probably irrelevant. (This assumes the full mesh topology, which with this hardware is not scalable over 3 nodes, might be to like 5 with somewhat more expensive consumer grade HW and is not really scalable to more than 8 nodes due to the sheer amount of cabling required)
Intel NICs are 5-10x that price. I'm not sure why, but my suspicion is that it has to do with driver support and/or interoperability.
mellanox is/was quite good at getting code upstreamed
maybe I need to do my own blog post about my pile of computers...
Yes, several of us love to read about that! I haven't switched to 10 Gbit/s yet...
Own three, can confirm for Windows and Linux.
The cheapest one I found that others related had ASPM actually working was the Intel X710, and those are much more expensive than the ConnectX-3.
* 2x25G throughput is higher than 40G,
* 25G latency is lower than 40G,
* you can use 25G ports as 10G ports, and
* you can use DACs to connect 4x25G <=> 100G
That last point is particularly relevant given the existence of switches like the Mikrotik CRS504, providing 4x100G ports on 25W.
if you run all older mellanox gear the cx3 can do the kinda nonstandard 56gbe as well
I am still on 1gbe... I guess I don't transfer anything bigger than a few GiB time to time.
you can see them all in the datasheet, I believe fcbt is the one with all the stuff enabled
https://www.eia.gov/electricity/monthly/epm_table_grapher.ph...
Also, in case of server hardware, it is excessive over basic tier electricity consumption, so it will hit those 50c/kwh.
The first column is residential customers.
My current bill for initial tier is: 42c/kwh delivery + 15c/kwh generation, and even higher over initial consumption.
I guess I am not "ultimate" customer.
The reason you don't pay the price listed on that page is because you are only one customer. You are not the average customer.
Also, there were several hikes since Oct in my understanding.
Looks like sales to "ultimate customers" means it excludes electricity that was not sold, electricity that was sold to resellers, energy lost, etc.
The form also collects information about revenue from delivery "and other related charges"
42c/kwh delivery sounds insane. I couldn't find much data about average delivery rates, but I plugged in a few counties here and it looks like many areas have delivery rates significantly lower than that: https://www.cpuc.ca.gov/RateComparison
That page says numbers are year old, I entered my zip there and it says delivery is 20c/kwh for me.
If you don't know the story: PG&E is delivery monopolist in CA, with exception of some places with local powerplants. It was found guilty in causing wildfires, lost in court and need to pay $XXB damages, which it now shifts to customers through multiple rate hikes in 2023 and more coming in 2024.
42c/kwh is initial tier cost, whatever is over limit will be charged at 50c/kwh for just delivery.
I don’t see why the same wouldn’t be possible for a NAS. Stuff like truenas can serve across multiple interfaces
I stopped reading at "California’s average residential electricity rate is 15.34 cents per kWh" and checked my electricity bill. Here in California, I'm paying 40.82 cents per kWh. The website https://www.electricitylocal.com/ seems wildly off.
Edit: I did read the entire article. Just stopped to check what I was paying.
That said, mentioning it is one thing (it makes the case for low power devices even stronger) but why stop reading anything after the first error you identify? That doesn't guarantee you only get accurate information it just guarantees you'll miss out on good information, like the cheap high speed connectivity the article is actually demonstrating.
I read the entire article.