Hacker News Reader

Comment by onlyrealcuzzo | Hacker News Reader

> I was able to get 70-80 MB/sec write speeds on the cluster, and 110 MB/sec read speeds. Not too bad, considering the entire thing's running over a 1 Gbps network. You can't really increase throughput due to the Pi's IO limitations—maybe in the next generation of Pi, we can get faster networking!

This isn't clear to me. What am I missing?

View on HN

110MB/s is gigabit. It’s limited to gigabit networking and only has 1Gbps out from the cluster board. So there’s no way to do an aggregate speed of more than 1Gbps/110MBs on this particular cluster board.

If each pis Ethernet was broken out individually and you used a 10G uplink switch or multiple 1G client ports then you could do better.

The write speed being lower than the read speed will be because writes have to be replicated to two other nodes in the ceph cluster (everything has 3 replicas) which are also sharing bandwidth on those same 1G links. Reads don’t need to replicate so can consume the full bandwidth.

So basically it’s all network limited for this use case. Needs a 2.5G uplink, LACP link aggregation or individual Ethernet ports to do better.

Which ICs would you use for this? Do you have something in mind?

I'm not sure about specific ICs, but if you took a look inside a Netgear GS110MX you'd have a good candidate IC.

Just a random search on Mouser, but something like the BCM53134O[1] as four 1GbE ports, and one 2.5GbE port. A bit pricier you have the BCM53156XU[2] with eight 1GbE ports and a 10G port for fiber.

Not my field so could be other, better parts.

[1]: https://eu.mouser.com/ProductDetail/Broadcom-Avago/BCM53134O...

[2]: https://eu.mouser.com/ProductDetail/Broadcom-Avago/BCM53156X...

The device has an onboard 8 port unmanaged gigabit switch. The two external ports are just switch ports and cannot be aggregated in any way. The entire cluster is therefore limited effectively to 1gbps throughput.

IMO it ruins the product utterly and completely. They should have integrated a switch IC similar to what's used in the netgear gs110mx which has 8 gigabit and 2 multi-gig interfaces.

It would be really cool if they could split out 2.5G networking to all the Pis, but with the current generation of Pi it only has one PCIe lane, so you'd have to add in a PCIe switch for each Pi if you still wanted those M.2 NVMe slots... that adds a lot of cost and complexity.

Failing that, a 2.5G external port would be the simplest way to make this thing more viable as a storage cluster board, but that would drive up the switch chip cost a bit (cutting into margins). So the last thing would be allowing management of the chip (I believe this Realtek chip does allow it, but it's not exposed anywhere), so you could do link aggregation... but that's not possible here either. So yeah, the 1 Gbps is a bummer. Still fun for experimentation, and very niche production use cases, but less useful generally.

110 Megabits == 880 Megabits, which is approaching the top speed of the network interface, which is the main bottle neck. A board with more IO, like the rk3568 which has 2x PCIe 2 lanes and 2x PCIe 3 lanes, or a hypothetical rpi5, can deliver more throughput.

Reply on news.ycombinator.com