110MB/s is gigabit. It’s limited to gigabit networking and only has 1Gbps out from the cluster board. So there’s no way to do an aggregate speed of more than 1Gbps/110MBs on this particular cluster board.

If each pis Ethernet was broken out individually and you used a 10G uplink switch or multiple 1G client ports then you could do better.

The write speed being lower than the read speed will be because writes have to be replicated to two other nodes in the ceph cluster (everything has 3 replicas) which are also sharing bandwidth on those same 1G links. Reads don’t need to replicate so can consume the full bandwidth.

So basically it’s all network limited for this use case. Needs a 2.5G uplink, LACP link aggregation or individual Ethernet ports to do better.