Comment by fxtentacle | Hacker News Reader

Comment by fxtentacle | Hacker News Reader

Wholeheartedly agree. After my cloud storage costs exploded (mostly S3 egress over to Hetzner/OVH), I noticed that renting a 1GBit/s fiber connection to my office is actually quite affordable at $80 per month (in northern Germany).

Of course, it's not globally distributed and there is no fail-safe, but for the work we're doing, that is no issue. If we employees are offline, it doesn't matter if our tools are offline, too. And now that all AI storage is local anyway, building a GPU compute node is easy. I'm still waiting for 3090 prices to drop further, though, in contrast to the article. But I also went with Ryzon 5950 and Linux. I was positively surprised that 10G fiber networking is now down to $70 for a PCIe card + 20m cable kit. My workstation now has 1010MB/s 4k random write on the network filesystem. (We used SAMBA 3.1 and CIFS mounts)

I also grabbed the python/ubuntu package lists off Google Colab and created my own Docker to imitate it, and now data processing and AI training is fast (I always get the good GPU, no luck involved) and dirt cheap. Originally the idea was to run it on OVH, but I'm now also running it locally.

https://github.com/fxtentacle/ovh-colab-sagemaker-compatibil...

> I was positively surprised that 10G fiber networking is now down to $70 for a PCIe card + 20m cable kit.

Wow, that is surprising, maybe it's time I started upgrading my LAN...

I use Intel 82599ES SFP+ and a TL-SX3008F router. But let me warn you: Things are affordable, but NOT consumer-friendly. I needed to study the 500 page PDF manual and do basic link configuration through telnet via USB before I could connect to the router via Ethernet and use its web-GUI to finish the setup.

How's the routing performance on that switch? I'm really struggling finding a device to fit my needs ( routing, 6-8 ports, Ethernet, at least two 2.5/5/10G).

8x 10G SFP+ with a 160GB/s full duplex backplane. So far, it has passed all of my testing with flying colors.

Isn't that only for switching? Do you use it to route, cross-vlan or similar?

Actually if you have cat5 or cat6 base-T setup, you can get second hand dual port 10gbe pcie cards for 30 dollars each(!)

10GbE baseT switches are more expensive than fiber equivalent though.

In my case, cat6 cables and the added price of a 10G RJ-45 switch would have been more expensive than a 10G SPF+ switch and some twinax & fiber cables. Amazon has finished SPF-to-SPF assemblies for €10=$12. And for TP-Link, RJ-45 is like 1.5x the price of SPF+ equipment. Also, RJ-45 has a fixed minimum latency, due to it needing to support backwards-compatibility with 1G 100M etc. That means you need much larger send/transmit buffers to saturate the link as ping goes up. Especially if you need multiple hops, fiber is just faster. In my tests, 0.2ms vs 3ms roundtrip time for 10G SPF vs. 10G RJ-45.

> Especially if you need multiple hops, fiber is just faster. In my tests, 0.2ms vs 3ms roundtrip time for 10G SPF vs. 10G RJ-45

ime if your unloaded latency on copper ethernet exceeds 0.2ms per hop there is some form of powersaving involved

If I remember correctly, the IEEE 802.3an line coding overhead is around 2.6ms for each time you switch between SPF+ and RJ-45. That's in line with my measurements.

SPF+ switch to OM3 fiber to SPF+ PCIe card => 0.2ms

SPF+ switch to RJ-45 cable to RJ-45 PCIe card => around 3ms

pretty wild, considering my phone averages at 2.2ms ROUND trip time when pinging the router.

NAS-[10GbE]->switch-[1GbE]->asus router, 2 hops ping test: ``` yatli@yatao-nas ~ % ping 192.168.50.1 PING 192.168.50.1 (192.168.50.1) 56(84) bytes of data. 64 bytes from 192.168.50.1: icmp_seq=1 ttl=64 time=0.236 ms 64 bytes from 192.168.50.1: icmp_seq=2 ttl=64 time=0.195 ms 64 bytes from 192.168.50.1: icmp_seq=3 ttl=64 time=0.300 ms 64 bytes from 192.168.50.1: icmp_seq=4 ttl=64 time=0.271 ms 64 bytes from 192.168.50.1: icmp_seq=5 ttl=64 time=0.153 ms 64 bytes from 192.168.50.1: icmp_seq=6 ttl=64 time=0.216 ms 64 bytes from 192.168.50.1: icmp_seq=7 ttl=64 time=0.300 ms 64 bytes from 192.168.50.1: icmp_seq=8 ttl=64 time=0.165 ms 64 bytes from 192.168.50.1: icmp_seq=9 ttl=64 time=0.163 ms 64 bytes from 192.168.50.1: icmp_seq=10 ttl=64 time=0.265 ms 64 bytes from 192.168.50.1: icmp_seq=11 ttl=64 time=0.174 ms 64 bytes from 192.168.50.1: icmp_seq=12 ttl=64 time=0.272 ms 64 bytes from 192.168.50.1: icmp_seq=13 ttl=64 time=0.397 ms 64 bytes from 192.168.50.1: icmp_seq=14 ttl=64 time=0.256 ms ^C --- 192.168.50.1 ping statistics --- 14 packets transmitted, 14 received, 0% packet loss, time 13168ms rtt min/avg/max/mdev = 0.153/0.240/0.397/0.065 ms ```

OK it seems mistake was to put a RJ-45 transceiver into an SPF port, if your direct RJ-45 line is so much faster.

Yeah I think so. There might also be better converter modules. Anyway, my setup is caused by an oversight of not burying light pipes into the floor. Otherwise I guess I'll go with SFP+ too. I'm using TP-Link TL-ST1005 which is the cheapest 10GbE switch out there and it cost like $150. Also RJ-45 adapters/switches run anecdotally hotter than SFP+ ones.

either that, or the equipment is in awe because you mixed up its acronym

That's what I did. A few months ago, I bought 2 10GbE NICs off Ebay to connect my main desktop and basement server directly. I already had cat 6a running between them.

Some advice: make sure you know the form factor of your NICs. I accidentally bought FlexibleLOM cards. They look suspiciously like PCIe x8, but won't quite fit. FlexibleLOM to PCIe x8 adapters are cheap though.

Reply on news.ycombinator.com