25 Gigabit Linux internet router PC build | Hacker News Reader

25 Gigabit Linux internet router PC build

michael.stapelberg.ch

489 pointssecure347 comments

> I read on many different sites that AMD’s current CPUs beat Intel’s CPUs in terms of performance per watt. We can better achieve goals 2 and 3 (low noise and low power usage) by using fewer watts, so we’ll pick an AMD CPU and mainboard for this build.

Unfortunately many reviews are very misleading here. Zen 2/3 CPUs have good performance per Watt, that's true. But for a machine like this, which will be mostly idle, this is not the interesting metric and Zen 2/3 systems show that you can combine good perf/W with poor idle power consumption (which is not true for their monolithic APU brethren, which are used in laptops).

One of the biggest idle power hogs for these is the IO die, so make sure that XMP is disabled and the memory uses one of the slow JEDEC timings. This should be fine for a router. Check that the SoC/NB voltage is set to 1 V or less. Some boards set this higher. In the AMD CBS section of the board firmware there should be an item "SoC OC Mode" somewhere. Disable it. Some boards allow you to set a new PPT (package power target), but it's worth pointing out that values which are too low will make the CPU very slow because it essentially forces all cores to very low power states in order to meet the PPT since the CPU can't influence the baseline power (due to fabric and I/O die). The upside of using a reasonable PPT of e.g. 50-60 W is that you reduce power consumption if some errant task hogs the CPU.

These settings make a big difference, but only if the CPU is really idle. Even fairly light loads (e.g. on a desktop, moving the mouse on the background) has everything rev up. In deep idle (nothing running at all, no user interaction on a desktop) you might get a Zen 2/3 CPU down to around 20 Watts, but as soon as anything is happening at all we're straight back to the 40-70 W region.

Using an Intel system for this would have likely saved 10-20 W.

Yes I can unfortunately confirm this. Don't know how much has changed, but about a year ago my current employer evaluated about 10 desktop pcs from dell, lenovo, hp and the likes as all staff was supposed to get new machines. One of the important criteria was power consumption. The few AMD systems that were among the contestants had absolutely ridiculous idle consumption and weren't even considered any further.

At work people leave their computers and screens on (and unlocked most of the time)... (edit): I tried to educate but nobody really care...

Hence the need for lower idle power consumption

Maybe in your office, but when I was in the office we had a culture of pressing Win+L or the dedicated lock key on the keyboard and letting Windows turn off the display after a minute of being on the lock screen.

Yes I tried to explain both the energy and security reasons behind locking and hibernate/sleep.

That's what Group Policy is for.

At my job people tend to post silly things in slack, like an "I love ponies" giphy image, when they see someone else's computer unlocked and unattended.

Using local retail electricity prices, the idle power draw of 20-40W for the AMD CPUs comes out to $20-$40 per annum.

If THAT breaks the bank, run screaming from your workplace as fast as you possibly can. The bean counters can't count, and they're being penny wise and pound foolish in the worst possible way.

AMD CPUs run circles around anything Intel makes. Total performance, performance per core, price/performance, and performance per watt.

Giving all of that up for... $20? Seriously?

Must be a company that has the employees turn on their PC and then sit around doing absolutely nothing?

Don’t we need far more information before going off like this?

Off the top of my head I’d want to know how many workstations and typical workload.

Many offices are hundreds of workstations, mostly idling, and simply navigating multiple different SaaS Web Apps. In that case the primary metrics would seem to be price/performance (don’t need much performance) and idle power consumption.

I love the new AMD processors and they’re fantastic for heavy workloads. I’m thinking Intel is still well in the equation though when it comes to a typical workstation in an office.

> I’d want to know how many workstations and typical workload.

That sounds like: "Sure, we make a loss on every unit sold, but we'll make up for that with volume!"

If one workstation is better, a hundred workstations is a hundred times better.

> I’m thinking Intel is still well in the equation though when it comes to a typical workstation in an office.

Are they though? My "typical office applications" experience is that thanks to Electron they've become absolutely glacial, and better CPUs make a very noticeable difference even to ordinary staff. People rant and rave about how good the Apple M1 chips are for typical "productivity apps", which is directly comparable to the advantage AMD has over Intel.

As a random example of this, our virtual desktops in Azure are Intel-based. They're frustratingly slow. Switching them over to a largely identical AMD instance makes them instantly snappier and very usable.

PS: I had the opposite experience a decade ago with Citrix XenApp terminal servers. At the time, a few of our customers tried to save money by using inferior-but-cheaper AMD CPUs. Those servers were glacially slow, and there was nothing we could do to fix it.

How much does it cost to run new power through a room to handle “only” an extra 5 amps when you’re already maxed out? Without shutting work down?

How much does it costs to install another couple tons of AC when you are already having trouble keeping the room at 74°?

It’s easy to paint people as stupid when you think of things incrementally and ignore thresholds entirely. The bigger fool is the person who treats hardware and physical infrastructure as if they are elastic. They are not.

For starters, it's almost 30ct per kwh here. But admittedly I don't know what they pay being at 12k employees, size-wise. Most of these computers are being used for mundane office jobs, so I'm fairly certain idling is what these machines do for most of their life. As for performance, I'm still running a third gen i5 at my job from 2012 and actually doing dev work. It's still doing just fine. I think I'll upgrade this time, but I skipped the last 3 models out of laziness.

> Our network cards need PCIe 3.0, so that disqualifies 5 chipsets right away: only the A520, B550 and X570 chipsets remain.

No, the linked Wikipedia page only shows the PCIe lanes connected through the chipset and doesn't account for PCIe lanes provided directly from supported CPUs. X470 supports CPUs which have PCIe 3.0 and a board with x8/x8 mode like the ROG Strix X470-F https://www.asus.com/microsite/motherboard/AMD-X470/ should work just fine (and has no fan).

Thank you. I have added a note to the article with your correction :)

Very cool post. Thank you.

This article, and its many links, are helping noob me learn how to ask about I/O perf.

TechEmpower benchmarks HTTP servers. While contestants continue to improve, I've long been curious what the theoretical fastest HTTP server could be. https://www.techempower.com/benchmarks/

Basically, an update to the "C10K problem" for the year 2021. https://en.wikipedia.org/wiki/C10k_problem

A bit like how Daniel Lemire initially thought about parsing JSON. Paraphrasing: Why can't a JSON parser run at wire speed? "Parsing JSON quickly: early comparisons in the wild" https://lemire.me/blog/2019/03/02/parsing-json-quickly-early...

Also, I really appreciate including the sound and cooling considerations. It's just great seeing the process of system design accommodating (balancing) multiple goals. Bravo.

> Basically, an update to the "C10K problem" for the year 2021. https://en.wikipedia.org/wiki/C10k_problem

No, routing, natting and terminating connections are very different problems.

Especially if we consider different implementations.

The most surprising details was [1] from init7.

>Compatibility requirements There is no obligation on you to procure the hardware through us, and the hardware shown here is not the only possible hardware for you to use. There are also other compatible products, as long as the requisite «bi-di» fibre optic technology conforms to the following specifications (recommended: Flexoptix, more router information):

No Modem / ONT. Just a Router with compatible SFP optic. I wish more ISP do that. And not force me to use your crappy ONT or Wifi Router. They could of course go another router and provide actual decent ONT or WiFi Router. But the chance of happening, or they care about quality is slim.

[1] https://www.init7.net/en/internet/hardware/

A lot of fibre is GPON (a type of PON, passive optical network), a scheme where each customer receives data from multiple other customers as well as their own data, so it would be very insecure for them to allow customers to use their own ONT. If you can get AON (active optical networking), then likely they will allow BYO ONT. Note that AON requires more hardware that is more expensive on the ISP side.

I know there are lots of GPON Network that allows you to use your own ONT, and I think that is even by law in Germany where you could log in with username and password just like DSL. I know in GPON they are all shared but I never thought about the security aspect of it, so thanks for pointing that out I will dig a little bit deeper in to the subject.

Isn't the downstream traffic in GPON encrypted?!

I just looked that up, and you seem to be correct indeed, AES-128 is used.

In https://twitter.com/kuenzler/status/1397814747379154944 (German), Fredy Künzler from init7 talks about PON and why init7 is opposed to it.

He makes the calculation that from the ≈3000 CHF needed to connect a location with fiber, only 50 CHF difference come down to PON or AON. The advantage of dedicated lines (for all ISPs, too!) overweighs the financial savings in his reasoning.

That is a surprisingly small difference, although when you are building a greenfield last-mile fiber network for an entire country that small difference probably adds up to millions, which is probably why we got GPON in Australia.

I'm always amazed by people who don't bat an eye on the perspective of having a home server, sucking up electricity 24/7. In this case replacing a typical router (which consumes like a lightbulb), with a full-fledged PC (probably consuming like x100 the power).

I guess some people around the world have quite cheap utility bills! For me, it's either a Raspberry Pi type of power consumption, or else a server that only powers on when needed. But I haven't learned yet how to do the latter, if possible at all.

In fact this is a nice place to ask: how would you build a "something" that monitors the network for packets sent to powered-off machines, then somehow caches the request, powers the destination machine On, and finally lets the request continue to its target? Has this been tackled anywhere? There must be tons of people wanting a homeserver but living in places where electricity has a considerable cost...

There is a middle ground - either an Intel Atom or Intel Celeron based server - typically use around 10w and you still get niceties like SATA and even PCIe

Also plenty of slightly more capable laptop class SKUs that are in this ballpark.

Can you still find "desktop" motherboards to go with them?

A long time ago, I remember there being desktop MBs for the Pentium M.

For an always running thing, choosing a laptop motherboard is a nice way to guarantee all the power saving features will work properly.

Most desktop motherboards won't allow the PCI express bus to sleep for example, using an extra ~4 watts, $8 per year in eco-friendly countries. Across the ~10 years your NAS will probably sit in a cupboard, that's $80, which isn't an awful lot, but probably would have let you buy a bigger SSD or hard drive which would have had more utility.

That's true, but then again I wouldn't be comfortable with a single-disk NAS...

However, for running random VMs, etc, I think a laptop is quite a good proposition.

If you are OK with external USB drives, I did just that with my Atomic PI [1] (Atom-based, 35$ SBC, 3.5W idle, running Debian on its internal EMMC, with two external USB drivers in ZFS-mirror configuration).

[1] https://www.thanassis.space/atomicpi.html

You can also get rackmount servers in 1U format with Atom CPU's that support 4 3.5" drives, e.g. https://www.broadberry.co.uk/intel-atom-rackmount-servers/cy...

What a lot of other people don't realise:

- There is no such thing as a 25gbps home router.

What's currently on the market is a very serious overkill ever for 10gbps.

In terms of CPU it is not totally overkill if you're using an IPS/IDS, bare in mind you'll also be disabling most of the network card offloading in a full fledged firewall and that will ultimately result in consuming slightly more CPU cycles.

> I guess some people around the world have quite cheap utility bills!

Or they work at Google and don't really have to care.

Electricity is not particularly cheap in Switzerland, but not particularly expensive either (nothing like Germany for instance). If running a home lab is your hobby, why not. There are plenty of hobbies that are a lot more expensive.

The machine's clearly massive overkill for routing. Another commenter points out it typically runs around 50w - and the OP says they're also using it as a server in this thread.

FWIW: raspberry pis (the 4 series) sucks in 15w which is why it’s so bloody difficult to power them through standard USB power adapters (which go up to 12w).

So, not hundreds of times more power hungry, but definitely 2-3x

yep, which makes them a pain in the ass to efficiently cool down without active cooling (=fans)

My passively cooled RPi4 with 8GiB RAM + 1GiB/USB ethernet dongle working as a 1Gb router never goes above 50 degrees C (all-metal case from aliexpress).

Just to note that's peak power consumption. They idle between 1-4W, or if you're running a PoE HAT, 4-6W.

A RPi is not equivalent to what most would use as a COTS router, which includes a GbE switch and some kind of modem (DSL or cable). The latter on its own needs a few Watts.

On the contrary, OpenWRT and a good CM4-based board allows the Pi to run pretty well as a gigabit router :)

https://www.jeffgeerling.com/blog/2021/two-tiny-dual-gigabit...

That’s not much of a router with just two ports and no WiFi.

No CM4 and accompanying carrier board needed . A regular RPi4 with a USB3 gigabit adapter makes for a good gigabit home router. I have been using one as my home gigabit router for the last 14 months without any issues.

As another commenter already pointed out, this is peak power consumption. A Pi + external HDD runs me to 12 W max in live use. Less if the drive spins down.

The problem is running it 24x7, then a small 50w is still 438 Kwh each year. That's more than a 10% increase in yearly use for a typical two person household...

Now it's up to the person to decide whether increase in living cost is worth it, which... is kinda normal. I think I spend more money on other things that I take enjoyment from.

Sure, it's just that many people don't think about this impact because 50W sounds low.

Then like another poster said, it all depends on how much you pay for electricity and some people have quite low cost electricity.

For me that would be a $49 cost per year, or a cost of little over $4 per month.

Maybe having single bulky, but efficient server means other laptops/whatnot will be used less which will cancel out and be electricity negative at the end of the year.

How about people power this stuff with renewalbles like solar?

Aren't 60w lightbulbs still around?

Do you run your lightbulbs 24/7?

If so, you should get a LED alternative.

That's very roughly $100/year, which is < 1/6th of the cost of the _network cards_ in this build.

My workstation bursts to 600w+ for 15-20 minutes at a time when compiling, for comparison.

For developer like me, I'm actually considering this option, it would be great to have PC like this available for some docker stuff as well, ie. MSSQL instance for development (Azure MSSQL version that runs on M1 is shit slow to the point it's unusable for development). Maybe even some tests could be offloaded from dev machine to this one etc. Would be nice to have single place for backups, photo library etc.

Why not just use your development machine for development?

Also, why not use a remote server for deployment, testing, and building docker images? Most roles I have had offer such services (remote servers) as part of their costing so wouldn't cost you time, money, or effort.

I agree that having a single place to backup photos etc, is important. I use an external SSD for such a purpose as imo it's more useful to have it offline as I rarely add data and it is less likely to be comprised, e.g., it my machine was hacked.

>> Why not just use your development machine for development?

He said the M1 versions of MSSQL performs terribly

Do you know of any smaller scale hardware capable of routing 25Gbit/s?

EOLd mellanox sx6036 switches can be had for $200 on ebay.

That’s a switch though, not a router OP is building. And I doubt this heavy duty thing uses less power than OPs setup (and we’re taking in a thread complaining about power usage)

I agree. that it uses (relatively) lots of power (and is very noisy). but a managed switch might serve the guys router needs as well. depends what hes trying to do. Most people using a "router" inside the home, can probably get away with a managed switch. But another question was cost, so was trying to address that.

I think if the OP uses this only as a router it is indeed a waste of power.

What I would probably do is also use it as nas/workstation by using virtualization, SR-IOV is now pretty standard on these cards.

I’m using the machine also as a server, so it replaces 2 existing devices. In terms of power consumption, it’s likely only a small increase, if at all.

Also, the user can use his freedom to build and do whatever wished for. This tone of you shouldn't do this because some COTS is available sounds very commercially driven. People have the power to create another Google, we are not powerless.

My only rub with that is - shouldn't the router be only a router from a security perspective? Definitely combining servers for home use does make sense though.

Ideally yes. But with the resources needed for 25 Gbps, not using one machine for multiple purposes seems wasteful. The server only stores publically available data, though, so not a big deal from a security perspective.

What do you mean by resources required? expansion ports on the mb?

Yeah running your nas-bittorrent/firewall-router on the same OS is clearly not something I would do, especially that now you can use somrthing like proxmox for example and virtualize your pfsense/opnsense instance.

I am using a home server with the same CPU and I am using it for SQL, storage server and virtualization. In order to do that I changed RAM to 32GB DIMMs, 64GB in total was not enough and the CPU works just fine with 32GB DIMMS. The platform is quite limited by the number of available PCIe lanes, but without moving up to Threadripper (a lot more expensive) there is no better option, Intel is in the same place or worse. The good thing with Intel is that you can use a CPU with integrated graphics and save the PCIe lanes for the graphics card.

My previous build (Ryzen 2700, 65W) had a great feature until it was gone with a BIOS update: after installing everything it worked with the graphics card removed. As I always connected only remotely, it was not a problem but a benefit. You can try and see what happens.

Yeah, I have an overpowered router PC (mostly in the capabilites sense --its a low power passively-coolable Xeon), but it also acts as a flash NAS, and hosts a few other containerized services that I prefer are always-on, it's a wireguard endpoint, etc. I've got a separate sometimes-on box for other VMs and containers. That one hosts a windows VM with a VFIO-attached GPU which my living room tv plugs into. Altogether it's lots of computing power for home use but draws relatively little at the wall. I'm pretty happy where I landed in terms of overall utility versus TCO, using this sort of consolidated-hardware approach.

What kind of light bulb? :)

A typical consumer router will take up something like 5-10 watts. A PC will suck in 20-50. Remember that the PC will be a lot more powerful so it’ll spend most of its time with low CPU usage.

Say the worst case scenario: 45W difference. 45W * (24 * 30) / 1000 = 32.4 kW*h/month. At $0.10 kWh rate that’s $3.24/month, less than a cup of Starbucks coffee.

Add HDDs and 10 GBit/s hardware and you surely reach 60 watts or more idling. Many home labs also don't use the newest hardware but old server hardware (add 10 - 40 watt idle).

A NUC or other Laptop hardware on the other hand would be OK to run. Maybe SSD raids are (financially) feasible for everyday files in the near future.

You are lucky to pay only $0.10. Here (Germany) I pay 0.30 Euro per kW/h. About $0.36 dollars.

I turned off my home NAS a few months back and partially switched to VPS services (also because of the better connection).

> Many home labs also don't use the newest hardware but old server hardware (add 10 - 40 watt idle).

Yes, but switching to newer, more efficient hardware is not free, so you need to factor this in when considering a switch. Also, in case you don't only care about your wallet but also environmental impact it's getting even more complicated, since manufacturing your new shiny toy definitely is polluting the environment somewhere in Asia, gets shipped via ship burning horribly dirty fuel in international waters etc.

If you don't have any use for the old hardware, chances are it will end up burning on some open field somewhere in Africa after local recyclers already extracted the good stuff...

"your new shiny toy definitely is polluting the environment somewhere in Asia"

Thats true whether I use dropbox or my own server

> A typical consumer router will take up something like 5-10 watts.

You demonstrate a good worst case, but the article writer wants to use more than 10 gb/s so he can't actually use your typical router, he can have 15 gb/s with the MikroTik CCR2004-1G-12S which has an unknown idle W and a max around 50W.

Looking into the problem, I can't really determine why I should upgrade to 10 or 25 gb/s, but if I wanted to do so now I would rather buy components I could reuse than buy a router that will be inefficient for its entire service life.

Oh my I wish those $0.10 kWh... in here we have 3 prices through the day (depends on what time it is), and the cheapest one is already higher than that :)

Anyway I did the math and it would be $7/month. More than double your estimation, but still not horrendous. Although for that range of prices one might be able to find a managed instance machine in some cloud provider...

(edit: somewhere in the process I lost track of the fact this was a price _difference_ calculation, so it's adding $7 to whatever was already the cost with a more power efficient machine, which ofc. depends on number and type of HDs and other equipment)

The real difference is in latency: I tend to use the same operator on the fixed and mobile line, with Wireguard. SSH, SMB and Matrix respond almost as if you were in LAN.

If I add the cost for bandwidth and storage of a data center, then the economic choice is obvious.

Would it be efficient to store power when price is low?

He claims one of his goals is to be power efficient

“The PC consumes about 48W of power when idle (only management …”

https://www.eia.gov/tools/faqs/faq.php?id=97&t=3

“ How much electricity does an American home use? In 2019, the average annual electricity consumption for a U.S. residential utility customer was 10,649 kilowatthours (kWh), an average of about 877 kWh per month”

Ok so what am I missing here? In Germany a typical four person household is 2000kwh a year. Surely air conditioning can't make up for such a difference? Everybody already switched to electric cars?

I don't have the numbers, but air conditioning can suck up a LOT of energy. I'm pretty sure air conditioning would be (in our household) the single biggest energy user.

That's about 170 kWh/month. I'm in the US and use around 450-500 kWh/month in the summer, which is my time of minimal use because it is warm enough to not need heating and usually not so warm as to need air conditioning.

I'm curious why such a big difference. Here's what I'm using in a household of one.

1. Kitchen appliances: Fridge, microwave, toaster, oven, dishwasher. The first three are used daily. The oven and dishwasher once per week. I'll also occasionally use a bread machine and an electric kettle, maybe a couple times a month.

2. Washing machine and dryer. 4-6 times per month.

3. A 2017 27" iMac, a Raspberry Pi 3, cable modem, TP-Link A7 router, and two TP-Link SG108E switches. A USB hub and a case for a couple external SSDs. An external monitor.

4. A 55" LCD TV which is used a couple hours a day, and an A/V receiver which is always on but usually idle. A couple streaming boxes (Fire Stick 4K, Xfinity Flex).

5. iPhone, iPad, Apple Watch, and Surface Pro 4 chargers charging those devices.

6. Indoor lighting. It is almost all LED. The only exceptions are the lights in the fridge and dryer, one bulb each in the attic and crawl space that I forgot about when switching to LED.

7. An outdoor security light in back that is on overnight. I'm not sure about its power consumption, but these are typically under 100 W, which would be 24 kWh/month during the summer.

(I'm a bit puzzled by that light. I'm still using the same bulb that was there when I bought the house, giving the bulb on "on" time since I've owned it of around 56000 hours. That's quite a bit longer than expected for every kind of light bulb that I can think of that this might be. Only LED should approach that and definitely is not LED).

8. Well pump and water heater. The well pump runs maybe a couple times a day for maybe 5 minutes at a time, and would account for at most a few kWh/month. I know the water heater is a beast--but I think it only runs a little more frequently than once a day.

9. A couple box fans in windows at night to blow out hot air. They are 50 W each. Maybe 50 kWh/month.

10. Miscellaneous. Charger for rechargeable AA and AAA NiMH batteries. Electric toothbrush. A Google Home Mini as a kitchen timer. An Echo Dot to control lights. a Hue hub for the lights. Charger for the batteries for some cordless tools (drill, string trimmer, hedge trimmer) that are all used rarely.

My house is very atypical for my area of Michigan. It's all electric with a geothermal ground source heat pump for heating and cooling and electric water heaters. In addition it has (or rather had) two of these geothermal units and two water heaters for two separate spaces. We also have a plug-in hybrid vehicle that we charge at home.

Outdoor temperatures in the summer range from mid 70s F to low 100s F and humidity is almost always above 60%, hitting 90% for weeks at a time.

Our electric usage in all but the coldest months of the year is around 2200kwh per month. I expect this to go down somewhat because we just upgraded one of the geothermal units and replaced the other with a gas furnace and inverter AC unit.

It looks closer to 3.5Mwh:

http://shrinkthatfootprint.com/average-household-electricity...

But still that's far lower than the US at 11Mwh. Maybe A/C and home size accounts for most of it.

We're a family of 3 and on track for around 8Mwh this year. We only use AC on the hottest days (Southern Californian climate so it's quite manageable throughout the year), water and dryer are gas - but I have a homelab which could be optimized.

I got that figure from my last bill, they always add graphs and comparisons, like a bar chart displaying usage of a typical 1, 2, 3, 4 person household and then your usage. But maybe they show lower values here to get you to try and save more.

I'm in a two person household and usually clocking in at 700-900kwh, and that's with a 24/7 home server drawing 30W, a rpi for kodi that I keep running, dish washer, washing machine, induction stove. No ac, no dryer, no microwave, hot water and central heating with natural gas. Also no more desktop pc since around 2014.

Have you seen the size of an American refrigerator compared to European?

We use 2kwh a month here in August.

>"I'm always amazed by people who don't bat an eye on the perspective of having a home server, sucking up electricity 24/7"

Some people simply run business from home and it is legitimate business expense. For example in Toronto server consuming 200 Watt 24x7 comes to about $20/month. Not much to dwell about if you are making money as a business.

I have a computer in a room that’s generally otherwise too cold… so all that power is just useful heat dumped into the room.

If you pay a lot for electricity, you could get heat cheaper with a heat pump, gas furnace, or simply adding insulation on the roof/walls so any heat you do add lasts longer.

Re-insulating a house is a substantial cost that can take a decade to pay back in utility bills.

Yet spending $6 on draught excluders to keep the wind out can pay back in just a day or two.

It all depends on your baseline...

Albeit not as efficient as say a heat pump

Significantly cheaper though.

Cheaper to buy, but not to run

Commodity window-mount heat pumps which run off household power outlets can be had for a few hundred dollars, and are good for a minimum of a couple kilowatts of heat output.

Any computing solution which dumps a couple kilowatts of heat and didn’t fall off a truck would probably cost more than that.

Computers are fairly efficient heaters, but they’re a lot more expensive than heaters.

That depends. The heat might be used in a heat pump for all we know.

Well given that all but a tiny percentage of the energy consumed that’s not waste heat is either going into useful computing or keeping fans spinning (which you would have in any sort of HVAC solution), I’d say it’s super efficient.

I do remember a "smart NIC" making its rounds in the news a few years ago -- although "few" in this case might be >10. It was essentially a low-power machine in itself (with no Raspberry Pis in sight yet) that could finish an HTTP download or receive e-mails without waking up the host computer. But that's more high-level than the buffering/relaying of packets you are thinking of.

Edit: a bit weird, I wrote a very similar comment a year ago about this item. Now I feel slightly compelled to find the news source for it...

asus "killer nic", almost pure marketing wankery with little benefit https://www.pcgamer.com/motherboards-with-killer-network-ada...

"Application offloading" was one of the terms I tried when searching for it, and got this as a result - not the same thing, but similar: https://www.academia.edu/10225597/Putting_it_on_the_NIC_A_ca...

https://pcper.com/2007/03/bigfoot-networks-killer-nic-k1-and...

My home server is about 70W, meaning about 150€ a year (~0.20 eur/kWH). So about 12€ a month. For one of my biggest hobbies. It’s fun, but it also runs WireGuard, 3 NextCloud installs, Home Assistant and Mosquitto, 2 MineCraft servers, a FoundryVTT instance, samba, sabnzbd, Unify Controller, an Nginx static site, LibreSpeed, VaultWarden and soon a Django site. Oh and a virtual desktop (vnc) I can always leave running with stuff open.

Do I need all of that? Meh, but it’s not bad value for money-wise IMHO. Perhaps mostly because I just enjoy it.

Edit: of course the thing itself was also quite expensive but not much more than a decent NAS which I think is a must for many people anyway.

> but it also runs ... MineCraft servers

At least you have your priorities straight :P !

> (~0.20 eur/kWH)

Are such high electricity prices common in Europe? That's higher than it is in 49 of the 50 US states (assuming 0.20 EUR = 0.24 USD, and with Hawaii being the exception).

In Western Europe at least this is quite common. Mostly because politicians have discovered this as a great tax resource.

Electricity in the EU is really really expensive. That person is actually paying below average rates.

That's cheaper than California!

0.30 Euro per kW/h in Germany

I assume it varies a lot by region. 0.12 EUR/kWh for me (Finland, inc. transmission and tax).

I never thought about it, as far as I know it’s been stable like this for years now… Gas is getting more and more expensive though.

Yes we should stop chastising people like that. You have long hair, you use more water, shampoo and drying. You drink coffee how many wh for that cup and its content. We could always find something in others where they consume more. My neighbors keep all their lights on almost all the time in every room. The other one has AC on all day to full power even when it is cooler outside... So complaining that someone uses a 50w/70w device for their hobby... meh . However I still believe we and the manufacturers should work to reduce idle power and consumption in general.

Oh I never felt chastised… before…

I mean I agree with not wasting energy of course, but what we’re talking about here is really nothing compared to driving a car or making a couple of pots of tea a day, for example. I refuse to feel chastised!

> My neighbors keep all their lights on almost all the time in every room

Almost all the lights in my house are leds, even if I did this the cost would be negligible.

Yeah I think this is a thing of the old days, my father in law is also always on top of light usage. Once I calculated what a 3W led cost him if he let it run the whole year (about 6€) to forget it once in a while is really no reason to get cranky. Leaving a couple of 60W incandescent bulbs on for longer periods can increase a bill indeed, but with leds it’s really not worth being know as the guy that whines about the lights for, IMO.

I don’t really think it’s about cost for these people. There’s something about a needlessly glowing lightbulb that is just offensive to them.

Well, I agree that lights should just be turned off when not needed, it also kills their lifespan. But you don't have to be a nasty person about it :)

You can dynamically turn on and off things with MaaS and monitor power though most UPSs.

let me introduce you to "My PlayHouse" https://www.youtube.com/watch?v=Jhwj_aCEYc0&list=PLS2odYzlao... fun stuff starts at 5:00

> who don't bat an eye on the perspective of having a home server, sucking up electricity 24/7.

Maybe they need it? A router isn't a home server. But even if, why do you keep your power efficient router running 24/7? You could turn it off when you're not home to save even more energy.

> a full-fledged PC (probably consuming like x100 the power

Maybe do some research before claims like these. He states this pc idles at 48W. Please show me a router capable of handling 25gbit/s that consumes 0,5W at idle.

> how would you build a "something" that monitors the network for packets sent to powered-off machines, then somehow caches the request, powers the destination machine On, and finally lets the request continue to its target?

That's what "wake on LAN" is for.

> But even if, why do you keep your power efficient router running 24/7? You could turn it off when you're not home to save even more energy.

You jest, but I turn off my router for the night and each time I leave home for more than a day. Not just router to be exact, everything that's connected to a power strip goes down, as I turn off all the power strips. (Not OP.)

Why not just go all in and shut off at breakers?

Fridge.

Then don’t flip the fridge breaker.

Plenty places don't have a separate breaker for that.

Breakers are not made for very frequent on/off cycles. They won't last as long as a light switch for instance. I've never seen a light switch wear out, but I have seen breakers wear out (at which point they trip very easily, and eventually seem to just permanently fail open).

Not a fan of the "turning the power strips off" method. Power supply failures are the number one reason why expensive electronics turn to bricks and subjecting them to the large inrush current that happens on the primary side every time you do the mechanical switch thing is a great way to significantly accelerate that process. Penny wise, pound foolish kind of thing.

That's the primary method of turning embedded devices on and off though. :) During development it's done around the clock and noone bats an eye. Rule of thumb: if a device doesn't have a power button, it's fine to turn off using the power strip.

I'd assume because development process doesn't need to deal with repercussions of accelerated power supply degradation

What do you mean? Things that don’t have an in-built switch, are meant to be fine hot plugging and as such, controlling via a power strip is fine.

I have physical switches on or near every outlet. Anything that's not actively being used gets turned off. With the exception of my oven and hob, because those are nearly impossible to reach. I use roughly 2kWh/day. This approach probably doesn't make much sense if you're using an order of magnitude more power. But it makes sense for me.

Crazy. My wife and I used 619 kWh last month

We were at 5MWh last month, so it’s all relative. Don’t feel bad.

That is impressive! Did you do that on a 200A 120/240V service?

What is your load? Ac?

Not the person you were replying to, but my June statement lists 1,800 kWh. I'm in a two-bedroom apartment in Texas. The primary consumer of power by far is AC, which is kept at 68 degrees in the summer (partly for temperature, partly for humidity). My effective rate after all the fees was $0.11/kWh.

It’s 240V/400A split-phase service, PNW, around $0.11/kWh. Air conditioning is a big part of it when the temperatures are high. Also charging multiple bEV, a lot of 24x7 loads: half-dozen fridges, a couple large freezers, wine cellar, about 1kW of IT+AV equipment, etc.

Have you measured how much electricity a computer doing routing consumes? I bet it's a lot less than you think.

Also measure how much a good, high-bandwidth router uses. I bet that is a lot more than you think.

the "why do this" question for me comes about entirely because of the continual manual intervention that is needed for solutions like this. just not worth it, to me.

I mean I have and my router is on average drawing around 15W but it also isn't remotely close to being able to hit 25Gbps. I don't think 50W (apparently the draw of this router) is unrealistic for a 25Gbps router by any means.

Maybe they have money and they find that spending money on homelabbing is worth the effort. And really it isn't THAT expensive either. I for one don't bat an eye on my USD200 yearly homelab electricity cost.

Of course if one wishes to perfectly optimize everything then having a homelab might well wasteful.

Indeed every case is unique. My home lab runs about ~4kw all the time, which end up being about 2.8mWh/month. Here in Oregon that ends up costing $340/month.

This is a brand new house, and as soon as I get permits approved I'm adding 21kw of solar which will help offset that.

His starting requirements include one pci card for 25gb and another for 4 port 10gb. I don't think there's any low power way to do that. Any motherboard with enough slots, CPU power, and PCI lanes to handle all that aggregate bandwidth isn't going to be low power setup. If you underpowered it, then you might as well back off of 25gb/10gb.

So it's not really a home server. It just happens to be in a home.

45W is not really "sucking up electricity", it's really nothing compared to other things in the house. I don't think PC is consuming 100x more as well, less than 10x and the same or nothing when not used.

I put together a tiny server with power consumption in mind, it consumes 15W and has i3-8100, that's enough to run just about everything I can be bothered to run: kubernetes, owncast, homeassistant, pihole, etc.

Generally, a laptop or Intel Nuc will give you a good low power server, much better platforms for development than a PI.

Then if you decide you need a RAID array, that's a different ball game.

> In fact this is a nice place to ask: how would you build a "something" that monitors the network for packets sent to powered-off machines, then somehow caches the request, powers the destination machine On, and finally lets the request continue to its target? Has this been tackled anywhere? There must be tons of people wanting a homeserver but living in places where electricity has a considerable cost...

IP protocol does not guarantee delivery. So you don't have to cache request. Just power on the machine, client will retry sending the packets until the machine is powered on and can respond. Just make sure that there's no gap between networking available and http server is still starting on.

I have built such a gateway for SSH: it accepts the connection, powers on the target machine if needed, then forwards traffic.

https://github.com/stapelberg/zkj-nas-tools/blob/master/wolg...

The advantage over just catching any packets is that an SSH connection is authorized, so less noisy in terms of undesired wakeups.

Apple has this. It's called Boujour sleep proxy, and it allows a always-on device like a Airport or AppleTV to claim the IP address of a sleeping Mac, waking it with a WoL packet when traffic for that device comes in. It's probably pretty useful, but is annoying on a managed network, as it spams my arpwatch system every time the IP is transferred between devices.

I personally run two servers with a NAS 24/7 acting as media, ftp, and web server. My electric bill maybe increased a couple of dollars, but otherwise I haven't really noticed any significant increase. I'm based in the southern US

I’ve got a handful of computers running in a rack at home. Their total power? Around 400w.

I also have a dehumidifier running all the time in the basement. It’s power usage is 500w.

No one bats and eye at a heater or dehumidifier, but a computer and people get worried

I mean, I would bat an eye at a humidifier running 24/7 without any sort of automatic on/off based on humidity levels but maybe in your climate that’s more normal?

I agree with your sentiment. It is interesting to look at total power usage and power usage by device. I'm using IotaWatts to monitor every individual circuit in my house (over 200 of them), and it is easy to miss things that add up. With computers it is quite amazing the difference in power load based on cpu/disk load. When I run my PI calculator (which not only pegs CPU, but hits my SAN very hard) I see over a 1kw difference in my homelab power draw.

Sorry, your house has 200 circuits? Is every outlet on its own circuit or something? Maybe I’m biased because my house is old, but I’ve never heard of having that much copper in the walls.

A few different things drove up the circuit count:

(1) It is a large house ( over 10,000 sq ft ). (2) There are two power paths throughout the entire house - generator and non generator, and most panel locations have two panels for that reason. (3) There are two incoming services from the power company - A standard 400 amp service, and a second 200 amp TOU (Time of use) circuit that is primary used for EV charging. And there are 4 EV chargers, two in the upper garage and two in the lower garage. (4) There is a 16kVa UPS and circuits behind that UPS that go to a few places in the house including the server room.

Lots of cool pictures and info here: https://www.garagejournal.com/forum/threads/jeffs-mountain-s...

Holy smokes - that’s quite the house. Thank you for the reply!

> "No one bats and eye at a heater"

Then where did the sitcom style stereotype of Dad always turning the thermostat down come from?

"probably consuming like x100 the power" is greatly exaggerated.

A typical PC might have an idle power consumption of less than 50 W, while a very small computer with ARM little cores might have an idle power of 2 ... 3 W, so at most the power consumption ratio would be 20.

However even that is not realistic because a computer or appliance with less than 3 W power consumption will not be able to route 10 Gb/s or faster links and it will struggle even with multiple 1 Gb/s ports.

A dedicated router appliance able to route 10 Gb/s or faster links will probably have an idle power of around 10 W or even more.

For routing only 1 Gb/s links, you can use among the standard PC's a NUC-like model, which will have an idle power consumption between 5 W and 10 W, quite close to a dedicated appliance. This is actually what I am using for my own Internet router/firewall (which also runs many other services, e.g. DNS server and proxy, NTP, e-mail, HTTP server and proxy etc.) with 1 external 1 Gb/s port and 4 internal 1 Gb/s ports (4 of the 5 ports are made with USB to Ethernet adapters).

Some of the more recent NUC or similar computers have multiple Thunderbolt ports or 10 Gb/s USB ports, so those can be used to route multiple 10 Gb/s links (using Ethernet adapters), with an idle power of around 10 W, similar to any equivalent commercial routers.

For 25 Gb/s Ethernet links, a commercial router is unlikely to be much cheaper or to consume much less than a standard PC.

SFP requires less energy than RJ45, only 0.7W/port. The MikroTik 4-port CRS305-1G-4S+IN draws 10-18W.

That's not entirely true. SFP power consumption depends on the type of SFP. A 10Gbps DWDM SFP+ might draw 1.8W of power. The why for this is actually quite interesting: the lasers used in DWDM SFPs have much more stringent requirements for temperature stability to ensure the light emitted doesn't drift out of spec. In order to achieve that temperature stability, they use a built in peltier to pump heat away from the laser and control the temperature. They're quite the marvel of modern engineering!

It would have been fun to be a fly on the wall at Cisco when they first realized people were jamming devices with heaters in SPF sockets.

Cisco didn't "realize" this was happening, they specified, had manufactured, and sold the very SFPs the GP post is describing. SFP slots in routers are designed with this thermal load in mind.

Which is also why plugging a large amount of DWDM optics into a datacenter switch is a bad idea. Datacenter switches are _not_ designed with this in mind. You run into risks of both overheating the switch as well as overloading the PSUs. A small number of high-power optics ain't gonna break the switch though.

And: 10Gbase-T SFPs have horrendous power consumption, even worse than DWDM SFPs. At these speeds, the signal over copper is mostly mush, and the PHY contains a none-too-trivial analog & digital signal processor. Which, again, is where limitations for 10Gbase-T SFP usage come from. If at all possible, avoid this shit — there's absolutely no reason to have 10Gbase-T inside a rack, for example. Just use DAC cables or SR optics.

I'm being a bit glib, but from the point of view of someone who used SFP optics once in a while, it seemed like the whole "standard" was a bit of a mess.

If it was known that vendors would try to push their thermals past what the device cage would support, why not address that better?

The general impression I was given is that thermal loads were one of the primary reasons given for vendor-locking pluggable optics, and that the original SFP spec didn't even address it.

Except no one is using DWDM for homelabs, as it's usually used for longer distances. The peltier cooler is definitely cool, though.

True. On general principle I would recommend folks invest in single mode rather than multimode fiber for permanent installs, as the price delta on SFPs is low enough these days. Multimode is a complete pain as it needs to be upgraded for higher speeds every decade. If it's just a couple of patch cables in a home lab, it doesn't matter, but if it's run through a wall...

It's also a real laser versus a many sfp+ gbics being LED light source now

> MikroTik 4-port CRS305-1G-4S+IN

I have two of these at home, they're neat little bits of kit.

They're fanless and therefore silent, which is fine until you realise you want to do 10GbE over existing copper cables with something like Mikrotik's S+RJ10 adapter. Then the temps start to rise...

So, I've decided to go completely to fibre, even if it means opening up the walls of the house. Just bought a job lot of used ConnectX-3 cards off ebay.

> ConnectX-3 cards off ebay.

Those cards are so good. I got a few after seeing them rated in r/homelab and haven’t looked back. The Synology took one and it was about 1/10th the price the official Synology one was.

> Those cards are so good

Agreed. You can find them on eBay for ~$40 a pop, and they work flawlessly on Linux and Windows. I've made a few purchases from the seller "bitsquad" (ships from Kentucky), and he always seems to have Mellanox/Chelsio in stock.

How do you obtain custom lengths of fibre cables that work with ConnectX-3?

Use fs.com, their generic cables work with MikroTik no issue and they do custom orders.

The ConnectX-3 cards are not vendor locked in any way. (And even if they were, that'd only affect the SFP, not the cable.)

Another good tip is to use Direct Attach Cables (DAC), which have a much lower power consumption and save you the cost of 2 fiber optics as well. This only works on short distances of course, but between the MikroTik and a few devices that might very well be the case.

> For 25 Gb/s Ethernet links, a commercial router is unlikely to be much cheaper or to consume much less than a standard PC.

EDIT: I linked to a switch by mistake.

This router has 12 x 10G SFP+ and 2 x 25G SFP28 ports and costs $595

It appears to be routing between 13 to 39Gbps depending on the rule complexity.

https://mikrotik.com/product/ccr2004_1g_12s_2xs#fndtn-specif...

That‘s a switch, not a router. Much simpler hardware.

Here‘s a MikroTik Router with 28 Gbit/a throughput: https://mikrotik.com/product/CCR1036-8G-2SplusEM

$1300 and up to 73W

That's a bit of an odd example to choose.

This[0] has a $600 MSRP, with twelve 10Gbps ports and two 25Gbps ports. It's ~32W before you add the SFP+ modules, and upwards of 50W if you have all modules populated.

In the benchmarks they list, this can provide somewhere on the order of 13Gbps to 40Gbps of routing, depending on exactly what you're doing. (Smaller packets will lower these numbers, but if you care about bandwidth you're unlikely to be worried about smaller packets... at that point, you probably care more about kpps.)

[0]: https://mikrotik.com/product/ccr2004_1g_12s_2xs

the benchmark you even link shows absolutely abysmal performance curves, dropping down into even half a gig perf for small packets single stream.

Calling any benchmark "abysmal" requires a point of comparison, and you have provided none... so it doesn't really encourage good discussion. What's the point of your comment? It seems to just be a way of insulting someone else's product.

Which comparably priced router are you thinking of that has a "non-abysmal" performance curve for small packet bandwidth?

Fair. It was the first router I found on their site with listed throughout >= 25 Gbit/s.

This is a switch and switching is hardware accelerated. While you can theoretically route with this this, it is incapable of routing at gigabit speeds, let alone 10 gigabit speeds, once you need even a few rules.

Actually that switch has hardware accelerated routing as well with RouterOS 7 IIRC. Take a look at the test results, for large packets that router will still do over 10G throughput with 25 ip filter rules. With no queues or rules it'll do routing just as fast as switching.

Edit: Looks like he changed from the switch to a separate router but look at the change log for the latest R7 release

https://mikrotik.com/download/changelogs/development-release...

>!) added L3 HW support for all CRS3xx devices;

So you get L3 even for this monster. https://mikrotik.com/product/crs326_24s_2q_rm

Interesting. I had only looked at RouterOS 6. I have had trouble with mikrotik software before though. (I have a CSS 610 and a lot of its core features were completely broken before)

> After my previous disappointment with the MikroTik CCR2004, I decided to try a custom PC build.

The article's author considered that exact router and rejected it.

He goes into greater detail here: https://michael.stapelberg.ch/posts/2021-05-28-configured-an...

For some perspective, this is a nice list of many household appliances and their power usage: https://www.daftlogic.com/information-appliance-power-consum...

Yes. In fact, apple’s AirPort Extreme units do that for macs in the house though their zeroconf networking. On sleep, the airport borrows the address and ARPs it. On traffic to that IP, it holds it, sends a special wake packet to the sleeping nic, and then re-sends the packet to with the real MAC, and the waking machine picks it up.

I run just plex and pi hole. Money I save with plex that I don’t need a million different services and having no ads at home is well worth the price. One day my friends went out and I couldn’t go. They got all drunk and shared how much they rely on my plex and by the time I woke up I had close to $1k in PayPal and a note that said “for an upgrade and your time” so honestly I probably making out on the deal. Went from a haswell pentium to a ninth gen i3. 8 gigs to 16. SSD for the root drive. 8 TB staging server before going to google drive. $20 a month for unlimited google drive is way more than the power to run the server. But I have about 80TB on there. Can’t store that for less than $20 a month no matter how you slice it.

I've been looking at the offers of big providers and the average seems to be pretty much standard: 2 TB for $10/month. All across Dropbox, Google, Apple, even Microsoft (the individual plan) are in the same ballpark.

Google drive enterprise for $20 a month for unlimited is the best I’ve found. Rclone and encfs to mount and send the content. Almost 0 issues.

sounds intriguing for backup purposes. I was not able to find this pricing online (without contacting sales), would you have a link for this?

I've been using pcloud which has a lifetime plan which is a pretty good deal. I recommend waiting until black friday or August 1 (Swiss national day; they're a CH-based company) when they severely drop the price.

I’ve read that it costs them $2 to store a single TB a month. Which makes sense. $4 -> $10 is a reasonable markup. I store 80 TB for $20. They lose money on me.

Does Google check or care what is on there? That would make me nervous.

I use encfs. Big scary warning on Ubuntu when you install it that it’s not 100% fool proof but for storing tv and movies in the cloud I think it’s fine. Best part is it’s totally seemless on my end. Uploads encrypted and my view is unencrypted.

Thanks.

bwahahaha, so they can't even hash similar common files and save the storage across users.

I have a HomeKit power plug that is able to measure the power usage. It sits before most of my living room devices including:

2x OG HomePods (Standby) - 55" Philips OLED (Standby) - Hue Bridge 2. Gen (Running) - ATV 4K 1.Gen (Standby) - Netatmo Weather Station (Running) - Linksys MR8300 OpenWrt 21.02 (Running) - Home Server (i5-6600, 16GB DDR3, 2x 256GB 850 EVO, 1Gb USB3 Ethernet Adapter as second NIC, Running)

Reading with all devices combined is 35-40W.

I mean if it's a problem just solve it. Yes, the folks in texas who have to pay 10x for electricity probably shouldn't get in if thuy can't afford it. Also, get solar/wind if you are concerned aboit the impact.

Many of these things are really competing with a $20/mo VPS so a full PC that's on 24/7 is still cheaper. Yes you could go further: use portable slim apps that fit on ARM SBCs but then you have to re-do your server config and learn a new app.

Plus, at least for me, my PC is on 24/7 anyway.

Why not? Every one needs a hobby.

Another benefit to Stapelberg's noodling is setting expectations.

Imagine you're bootstrapping a municipal ISP or mid-sized org. Dealing with vendors and products for the first time. What's reasonable? Who knows? Having these projects be a reality check is awesome.

I have a symmetric 10 gbps connection at home. I have an EPYC machine that plays host to a number of virtual machine guests for various tasks. One of them is running opnsense to be my router. It's not fast enough to route at line speed, but it's close enough that I don't care. A consumer router or a pi or something would be far slower, and I would start to care.

Having a tolerably powerful computer doing this means that it's also my web server for several sites, stores local backups and handles offsite backups, acts as my print server, and hosts a Windows virtual machine for using proprietary software (e.g., for my label printers or firmware updates for random widgets like my Lutron light system).

Quiescent appears to be something in the neighborhood of 40 watts. It's not nothing, but it's acceptable, especially for the utility. I don't pay for a VPS because I have the bandwidth and the capacity to self-host everything I want. I spend $8/mo in electricity to run that machine, and that will drop to zero when my solar array and battery become functional.

If the power consumption were 5x worse I would probably not have gone this route.

I have decade old hardware that is certainly inefficient, but my services hosted have taught me so much it's not even worth the comparison. Yes, electricity is cheap in America but we all choose to spend money on our hobbies as we please.

I'll bet there's a few nitpicks with your hobbies where you trade a lot of time/money/something that I don't necessarily understand. And that's okay.

When I was younger, 100W was a light bulb. We’ve come a long way. :-)

I've used managed power strips for data center applications in the past. The strips have individually addressable power ports and a web interface, telnet, as well as SNMP support. They've been around for decades and are the solution to your problem of needing to remotely manage the power state of power hungry devices.

If you set your servers to always power on after power is restored you can control them with that device.

There's also Wake On Lan (WoL) support in a lot of systems, where you can use a correctly crafted "magic frame" to wake up any machine that received it.

There is such a thing as wake-on-lan

https://en.wikipedia.org/wiki/Wake-on-LAN

My home server using a very similar configuration (same CPU, a lot less potent network) is what enables me to work from home, making a living, so the cost of electricity is compensated by the gas saved on commuting. I am also in process of adding solar to my house, covering a lot more than this server.

If people use old servers in the kilowatt range consumption for fooling around having a server in the basement is questionable, building something with low power in mind is nothing to complain about.

My home has a 0.5-0.7kWh idle load, with the varying 0.2kWh being from the refrigerators cycling. I tried shutting down my NAS and Home server and it was less than 0.1kWh.

I'm at a loss to explain exactly what is drawing 0.4kWh of power but my neighbor has a similar load on his. I'm suspecting all of the fancy and useless motion/astro light switches installed are partially to blame but there's no easy way to verify that.

My solar panels on my roof generate about 10KW/h more than I use. I can bank credits with the grid, but can never cash them out, so electricity is basically free for me.

My Netgear access point consumes 7-8watts. My home server, with 2*12TB drives, 2*3TB, and 1nvme, which also acts as NVR (for POE cameras), recording 24x7, and which also acts as owncloud server (and few other things) consumes 40-50watts (and it includes a 10gbps SFP+ fiber). It also includes a wireless card and acts as an access point. It also runs a few VMS, continuous integration server, pihole and other stuff.

This is not x100, it's x7 times. And the utility is much much higher.

Before, i was using amd Apu, and it consumed 20-30 watts, but did not support AES-NI, which made the disk access limited to 60MB/s :-(

It's way cheaper to run everything at home than paying for the cloud, even if u include electricity cost.

It's about $50/year for electricity (1 watt 24/7 =~ $1 per year)

It costs about the same as a Netflix subscription? The people that do this see it as a hobby, so it’s worth spending some money on.

A Netgear Nighthawk R7000 idles at 8W, max 34W. That will get you 1 Gigabit.

Bleeding edge CPUs from Intel and AMD do guzzle power when run at full load, but they also improve in power per IPS, and have added dynamic underclocking, so they also run

Compare the TDP of a Pentium 4 from 2000 to a Ryzen 5 3600 or an i7 7700K (These TDPs are supposedly max power output, but take them with a nugget of salt)

https://en.wikipedia.org/wiki/List_of_CPU_power_dissipation_...

Gotta update that power intuition!

What is the practical consumer (or even SME) use of a 10 or 25G connection?

As a consumer I’d definitely get it for the fun of it, but what is the point in reality?

We’ve just launched an ISP a couple of weeks ago here in Europe. Peak bandwidth use is about 3mbps per customer, and customers will download the same amount of data regardless of their plan.

I acknowledge this could be lack of imagination on my part, but the progression of bandwidth availability seems to be wildly outstripping demand (at least when it comes to fibre deployments).

Perhaps someone living in SF with a 25G connection can disabuse me of this notion.

> What is the practical consumer (or even SME) use of a 10 or 25G connection?

Remote file storage, leasing servers for things like icecream, or basic HPC, self-hosting for a small business, even a small DC can run off it.

Remote file storage I can just about see. But most consumers are streaming content (certainly TV, but possibly increasingly games), so where are all these multi-gigabyte files coming from on average 20/30-year-olds laptop/iPad? I don’t think people are shooting that much video, are they?

Renting storage, and accessing it over 10gbps at home may be cheaper than building your own multi-tb storage rack.

Absolutely. But from what I’m seeing, the average consumer’s need for storage space is decreasing not increasing.

Average consumer? Internet companies are responsible for a few percents of the labour market in some parts of the world.

Not so few people genuinely need gigabit+ internet for work.

is this the icecream you speak of? https://github.com/icecc/icecream I've never heard of it so I was intrigued.

Yes, so far a tool in a way better condition than distcc, though both are very poorly maintained.

At what speed do you start to run into limits on the server side when it comes to remote file storage?

Internally:

Every once in a while, I move VM images between machines. Even at 1Gbps, it's a drag to sit there and watch them transfer. Not really a big deal, but it'd be nice to see them move faster.

Externally:

I back up my Google Drive periodically. 500GB takes a while to transfer.

I agree, and this is totally the kind of thing I would do too. But I don’t think this represents average consumer behaviour.

> I back up my Google Drive periodically. 500GB takes a while to transfer.

Do you download the whole thing each time, vs just changed files?

Either way, (scheduled!) periodic backups sounds like a thing that could happen over trough (well, while you're sleeping), in which case a 200Mbps connection would be more than adequate for your use case (~5.6 hours for 500GB).

The numbers obviously scale linearly with the size of your download (a common anecdote I hear re: people filling up TBs of hard drives is via lots of RAW photos), so in that scenario, you'd need to transfer 30-40TB overnight (8 hours) in order to saturate a 10G uplink; you'd likely saturate hard drive write speeds first, not to mention you'd need multiple HDDs/SSDs connected to even store that much data in the first place.

It was a cinch when I had an always-on Windows box. I'd make a request via Takeout, save it on my Drive, and the Backup and Sync app would automatically download it to my local copy of the drive. But that box is now running Ubuntu, and I haven't gotten around to replacing Backup and Sync with one of the other (non-free, non-Google) options.

I would be very surprised if you could even break 2.5 gbps towards google drive. Saturating 25 gbps to anything the peer out of country would be basically impossible most places.

I can't imagine ANY reason that a private household would actually need a 10G or 25G connection unless it's doubling as a workplace and they're hosting servers. Even that seems like a temporary stop-gap situation, get some rackspace in a datacenter already!

Maybe they're not selling enough 25G to their business customers and they're trying to get private users onto 25G?

I can see that. Certainly if they already have the infrastructure deployed then a residential usage pattern on 10G is going to look identical to that on 1G (and I’d wager identical to that on 100M in most cases). In which case they can get more recurring revenue for likely only an increased one-off cost.

When I clicked through to the provider it looked like the only difference between 1/10/25G was the setup cost. But I might have mis-read.

Init7,OP’s isp is a non profit

No, they are a regular for profit company.

Regardless, their monthly (annual actually) fees are the same for 25G/25G as for 1G/1G. Only the setup fees are higher, which they attribute to more expensive optics. 10G/10G is completely identically priced to 1G/1G.

For download?

Working from home with large media assets of any sort.

For upload? Yes, I need ANYTHING faster than Comcast's BS so I can actually do any work OR even backups without killing my ability to do other things.

Right, that's totally reasonable, it sounds like you need an SLA (service level agreement) from you ISP, like a business-class service at 1G up/down.

Bu can you really make use of 25G or 10G service to the point where 1G is unacceptably slow? Probably not. It would strongly depend on the bandwidth of whatever you're connecting to.

As a deluxe nice-to-have with bragging rights, sure, but it won't make an operational difference except in very specific scenarios that almost no household would actually need.

I have 10G in Tokyo and I think it’s about $60 USD per month vs $40 for 2G. It’s definitely overkill but it sure is nice to download a 60GB steam game in a minute or two.

The problem with 10g is that most servers aren’t on 10g yet and if you’re transiting a peering connection of a smaller ASN at some point, even that is likely not 10g

Agreed, I struggle to get anywhere close to 1gbit/sec apart from Steam.

This hardly is relevant to your internet speeds with steam downloads. It is much more CPU dependent when you are operating at such high speeds.

A modern CPU + NIC combination will be twiddling its thumbs on a bulk download at 10G. There's a whole sleuth of optimization and offload features between the NIC and the CPU, and they work particularly well with a low number of high bandwidth data streams.

The real problem starts when you're doing lots of small packets all over the place. Which is not something you'll likely run into at home, ever.

No, it's still nothing to do with that for steam. Its their compression.

What do you use for a router? I've got 10G (price difference from 1G was even less percentage then yours) internet myself but currently just use an RB4110 as router which is the primary bottle neck. Would be nice with something reasonably priced and sized which can route 10G or close to 10G with NAT so I could utilize my uplink a bit more.

I’m still waiting for a reasonable prosumer one to come out. I’m using an intel 2 port NIC in my desktop (so only 2 devices including the desktop get 10G), and everything else is 1G.

I work from home and regularly need to download large files from work servers. Several months ago I noticed that my internal network was 100Mb/s, which was dreadful. Downloads of 500MiB files would take tens of minutes, because Tomcat would drop connections on such a slow download. And even without dropped connections it generally took a long time to download anything, and Zoom call quality was regularly degraded (I sometimes heard my colleagues as androids and vice versa). I fixed the issue and now I get the full 700M/s that my ISP offers (and 1G/s internally over a Netgear switch for NFS transfers). I could get a 1G/s, but it would cost me more and 700M/s is fine for the most time. But sometimes to get a shorter feedback loop on what I'm doing I could imagine getting a 5G/s. I think it would be optimal. But 10G is clearly overkill.

"Zoom call quality was regularly degraded"

It sounds like something else was dodgy on your previous network, surely Zoom cannot use up 100mb/s connection

When one side sends data at 1Gb/s and the other receives at 100Mb/s, it's worse than if the first side sent data at 100Mb/s. Here's a description from Julia Evans: <https://jvns.ca/blog/2018/07/12/netdev-day-2--moving-away-fr...>

Basically my ISP router would receive data at full speed, while internally I would have a lot of dropped packets.

Additionally, I usually use the web browser to view Jira or copy files over NFS during calls, and I'm sure there are some background apps contributing to the traffic as well.

The one thing that fixed all my problems was reconfiguring the network driver to set 1G bandwidth on my NIC instead of the 100M that it set automatically. Yeah, Linux Desktop problems...

> Basically my ISP router would receive data at full speed, while internally I would have a lot of dropped packets.

This is not how this works. The other end of your zoom call doesn't go "hey the person I'm talking to is on a 700mbits line, so that's how much data I'll send." By that logic you could extend this to the network of your isp, which is probably 10 or 100gbits, so you should have a lot of packet loss where the transition from 10gbits to 700mbits happens. And it's not even clear how the other peer could even know how fast your internet connection is, or the internal network of your ISP. That is what congestion control is for. Your peer can't know your modem's connection speed nor your LAN's speed. Software will simply observe how fast it can send data before packet loss starts to happen. This obviously requires that the there is some sort of feedback by the peer you're talking to. If you're using TCP, the OS will do that for you for free. With UDP, you have to implement that logic yourself in your application. And that's why it doesn't matter where the bottleneck is, it could very well be somewhere "in the middle" between two ISPs.

> I noticed that my internal network was 100Mb/s, which was dreadful. Downloads of 500MiB files would take tens of minutes, because Tomcat would drop connections on such a slow download. And even without dropped connections it generally took a long time to download anything, and Zoom call quality was regularly degraded.

Was you network setup broken? Did your firewall block ICMP? Ancient servers like tomcat and apache should handle slow networks in stride, even with modern updates - and 500MB at 100mbps is 40 seconds?

Now I can't imagine running 100mbps today for a home network... So upgrading makes sense. But your problems doesn't sound like they were caused by your lan speed.

In my case it was either 1G or 10G (no in-between), price difference between 1 and 10 wasn't huge, most of the cost was installing the actual fibre optics in the house.

10 Gbps can give you some advantage when downloading big games from a network that can support it on high end hardware.

Games are competing for "who can eat the most storage", it seems, and at 231 GB for CoD:MW (https://gamerant.com/pc-games-file-size-hd-space-biggest-hug...), you'll wait over half an hour before you can play with "only" a 1 Gbit connection.

The bigger the game, the bigger the incentive to just re-download when needed if you can do it fast, since your storage is limited.

The 10 -> 25 Gbps step is definitely just bragging rights, which I'd call well deserved given the lengths you have to go to to actually reach those speeds.

If anyone is interested in building 10Gb router on a budget you can buy Mellanox OCP NIC with an adapter from OCP to PCIe for around 5~30 USD ( depend on your location )[0]. I recently build one 10G router with 4x10Gb, 2x1Gb ports for around 200 USD using second hand PC.

[0] https://www.youtube.com/watch?v=hkf3PkuKdOA

I spent a couple of hours last week trying to set up a fiber connection in our new office. Since I didn't want to use consumer hardware in an office setting (and I already had a router that was compatible), I opted for getting my own hardware (a media converter).

I had _some_ prior knowledge of networking, but had never messed around with VLAN and MTUs before. Luckily I found this[1] gist for a comparable setup that saved me. I'm still only getting 350Mbps where I should be getting 1Gbps, but I _think_ that's just due to the bad networking chip in my cheap chromebook. When I move all my stuff to the office, we'll see what we can top out at...

All this is to say, I definitely respect the effort the author put into this and it pains me to find out that my new SFP setup is already obsolete :).

[1]: https://gist.github.com/Ruben-E/abb9a4a872a7c4ffff058ae291ef...

One thing I've run into when building my own 10gbps router that I didn't see mentioned - you probably need to disable power saving / frequency scaling. The handling of packets is done in software, so when you start a download and the flood of initial data comes in at 10gbps, your CPU will still be running at 600 MHz and unable to keep up. It ramps up within 5-50 milliseconds depending on CPU, but for a TCP download the sender will have already interpreted the dropped packets as congestion and reduced their upload rate.

You can monitor this with ethtool stats to see how many packets the NIC dropped due to host buffers being full.

In none of my tests did I need to change power saving settings. My iperf3 transfers stayed at 10 Gbps throughout.

You need to check your dropped packet statistics to know for sure. iperf3 TCP over LAN will not see throughput affected by a few dropped packets, a stream 100ms away may take longer to recover. High bandwidth UDP testing with iperf3 should also expose this, the first measurement will usually see dropped packets.

You won't see the impact because you're testing over something with very low latency.

He was only using 15% of the CPU at max speed

You can't measure the CPU at 1 second granularity to see this problem, in fact measuring the CPU at the sampling rate required may be enough to raise the CPU frequency :).

Was hoping for similar but 10 gbit. Anyone know a nice small/quiet linux box with 2x10 gbit for use as a router?

I have a 8 port x 1gbit ubiquiti router, but ubiquiti seems to be going downhill, and 8 port x 10gbit routers are pretty expensive. My plan was a 2 port router and use vlan tagging to a 8+ port 10g switch. That way I can have separate networks for trusted ethernet, trusted wifi, untrusted ethernet, and untrusted wifi.

AsRock makes an X570 ITX board with 2x10gig ethernet, the X570D4I-2T. Though at $500 for the board alone the resulting system is not going to be particularly cheap.

Look for an old but not outdated enterprise switch and grab a refurb. Most very large enterprises redo the network on roughly a 7 year cycle and in 2014 10g uplinks with tons of 1g devices was very standard. E.g. for less than 100$ you can usually pick up an ers 4826gts on ebay which has 24 ports of 10/100/1000 copper with POE+ capability and 2 1/10G SFP+ ports. It has hardware switching and routing which will perform better than software, particularly on latency.

As far as noise 1u devices are usually a bit whiney even when not pushing much air but that's usually fixed with 1 or 2 200mm PC fans.

You'll still likely need something with NAT, preferably hardware NAT, for the actual internet handoff. Some of this class enterprise device have NAT support (not the 4826) but that'll usually be in CPU and perform not that great especially if you have like a gig connection. Or if you want to go pure software here any cheap mini PC with 2 gigabit Ethernet ports should be able to handle NAT reasonably for gig internet or less since it doesn't also have to do the 10g internal route/switch on top.

I would look at getting a Xeon-D board. They don't draw much, come in mini ITX, and mine have 2x10GbE plus 2x1GbE built in, for a price that's reasonable when you look at standalone NIC costs. The CPU is soldered on, so the cost includes that, too.

I would go with a used SuperMicro A1SAI-2750F motherboard. It comes with an Atom 8-core C2750 (20w TDP) and a single 8 lane PCIe slot, but doesn't need active cooling. Supports ECC ram too. There's a passive heatsink on the CPU that works fine. I believe that would keep up with 2x10gb, be quiet, and power efficient.

They are $200 or so used on eBay. So not terribly expensive for something that can push 10g. And it will fit in any Mini-ITX case.

Don't be put off by the "Atom" branding. The C2750 was no slouch for its heyday.

I'd probably go with SuperMicro X10SDV-4C-TLN2F. It already has 10GbE.

   Xeon D-1521
   2x 10GbE ports onboard
   6x SATA3 ports
   Accepts ECC RDIMM DDR4 (usually cheaper than UDIMM ECC)
   1x PCIe 3.0 x16
   1x M.2 slot

There is one on ebay right now for $230 USD shipped.

Also, if you're going for lower power consumption I believe you can disable the IPMI and the 2nd 10GbE port if you don't need it. There are other ways to reduce consumption too. Turn off anything you can on the motherboard that you don't use and use higher density DIMM sticks (and less of them, if you can, e.g. 1x16GB rather than 2x8GB). Also worth paying attention to the PSU's efficiency curve and making sure your PSU is decent quality (reliable + 80+ gold).

>There is one on ebay right now for $230 USD shipped.

Ah, yeah, that's a better deal with the 2x onboard 10Gbe.

The cheap ones (including the one you mentioned) are useless because they have 10Gbase-T. You need SFP+ slots to hook up your provider's fiber.

Which is kinda sad too because implementing 10Gbase-T is more expensive to implement and more expensive to run (because the 10Gb-T PHY burns power.) Sadly, there's not enough knowledge about this going around.

yes, well you can get a cheap transceiver for that. You're still going to have to talk to something else and that all goes out the window if you have existing copper in your house or need to have wired devices (NAS, smart TV, xbox, literally anything at all). You probably can't even find a wifi AP that has SFP+ that doesn't cost you 3 arms and 5 legs.

> yes, well you can get a cheap transceiver for that.

Yes, exactly. I feel like you haven't finished your thought ;)

If you have a SFP+ slot, you can plug in a cheap SFP+ 10Gbase-T transceiver. Or a SFP+ fiber transceiver. Or a DAC cable.

If you have a 10Gbase-T port, you're stuck and need to buy and power a switch (or, recently, media converter) to get to anything else. And you're now wasting power for 2 10Gbase-T PHYs that are completely pointless for a very short link.

Of course you'll be using a bunch of 10Gbase-T for anything where you have existing copper runs that are good enough quality to actually run 10Gbase-T. I'm just saying (1) you shouldn't be using 10Gb-T for cabling things that are next to each other, and (2) you shouldn't be running copper cabling for new installations that you're building explicitly to get 10G over.

Note the Atom C2xxx series is the one where Intel f*cked up the LPC bus I/O drivers and it dies after a few years. Don't buy them used unless you're comfortable soldering a resistor to a clock line to extend its life.

This is fixed in the C0 stepping of the CPU, but finding out the CPU stepping is difficult even on new boards.

(cf. https://www.intel.com/content/dam/www/public/us/en/documents... - this is AVR54)

I'd love to get something simple that could handle a 10Gbit fiber7 connection and act as a 10Gbit Router & Switch in my home. My skills and patience in this area are a bit limited so going as deep as Michael is not something I could do with confidence. Narrowing down the hardware to something that is not only powerful enough, cheap enough, compatible with something like OPNsense and available to normal humans has kind of scared me off a bit.

Here I am having just had FTTH installed, but capped to a mere 160 Mbps. The main problem I am having is not the router (which handles such a "slow" speed fine) but the wiring throughout the house. It's all copper ethernet and because of the lengths and the fact that the runs must go parallel to some mains electricity cables, it tops out at 100 Mbps. I hope the poster has already put fibre around his house.

“The poster” (hey!) has put fiber around their house :)

https://michael.stapelberg.ch/posts/2020-08-09-fiber-link-ho...

https://michael.stapelberg.ch/posts/2021-05-16-home-network-...

Well, hate to tell you but you'll be limited 10Gb/s with these Intel cards because they require FEC.

You mean 10 Gbit/s per link? Even on the 25 Gbit/s card?

Where can I read more about FEC (forward error correction?) and how that affects link speeds?

Yes the transceivers recommended by Init7 (bidi-LR) do not support FEC and you'll be running in degraded mode (according to the controller datasheet), and I don't think this is something that can be achieved by coding of the transceivers.

init7 requirements are at https://www.init7.net/en/support/router-information/

The specific 25G optic is https://www.flexoptix.net/en/p-b1625g-10-ad.html?co10426=972... and indeed does not support FEC.

The remote end is a Cisco C9500-48Y4C, on which one can turn off FEC.

On the Intel side, I found https://www.intel.com/content/dam/www/public/us/en/documents..., which mentions that while No-FEC might result in “poor link quality” (sure, that’s why people use FEC), it does not say anything about degraded mode or limits to 10G.

By controller datasheet, I take it you’re referring to https://www.mouser.cn/datasheet/2/612/710_series_datasheet_v...

That document contains a state machine diagram listing 25G AUI‐RS, 25G AUI‐FC and 25G AUI‐N (in order) before the fallback to 10G SFI.

Did you misread or am I missing something?

In either case, I’ll try the card. If it won’t link at 25G, I can get a different one. Any recommendations? :)

The init7 CTO confirmed he’s running that same intel card and he’s getting 25 Gbit/s speeds: https://twitter.com/spale75/status/1414644121092952065

Great news! tbh I just assumed from past exprience with Intel NICs that it will negotiate to lower speed without FEC.

Personnaly I'll probably go with some cheap mellanox card on ebay :)

Kind of surprised there wasn't much attention paid to the software. My understanding is that with stuff like DPDK and fd.io you can get much better performance without going to extremes with your hardware. Netgate TNSR is one product that puts all the opensource pieces together to make it easy but it is all opensource software so should be usable by the average home user.

I was going to say that... with fd.io this should be pretty easy these days.

Author has written his own router software and probably wants to continue to develop it. They are much more pro than your normal average home user.

With a Xeon, a SmartNIC, and DPDK you can hit 100 Gbps.

The real story here is how great the internet service can be now in a civilized country if you don't let Comcast have monopolies and ban local internet cooperatives and coast along selling the same service for 15 years while raising prices.

25Gbps symmetric for about $70/month. That's significantly less than I pay Comcast for its crappy 900Mbps down/40Mbps up service.

At least in PNW the service isn’t 900/40 any more, it’s advertised as “1.2Gbps” and the profile actually seems to be 20% over that, I’ve seen 1.5Gbps or 170-180MB/sec downstream performance recently. Upstream is still <= 50Mbps and otherwise agree with your comment that the U.S. is falling behind in price-throughput on domestic internet services versus many other developed nations.

There have always been regional variations in this; I've had the same 1.2GBps plan in a few states now and it only ever got close to that number in the one where Fios was an option.

Comcast's only innovation in the last few years has been to add data caps to residential fiber service (my 300mb/s "1.2 GBps" is capped at 1.4TB/mo).

We’ve just been paying a $30/mo upcharge for unlimited data (removal of overage charges). Basically because it’s more predictable than the $10 for every 50GB overages. A few months we’ve had 4-5TB through the service and Comcast doesn’t seem to have slowed us down or charged more.

Let's please not further the idea that it's acceptable to quote only downstream speed when citing Internet speeds.

Doing that just leads to 20 more years of Comcast killing the concept of modern Internet service for Americans. Cite both numbers.

I consider only symmetric speeds real internet.

It's sad how difficult it is to get symmetric speeds in some parts of the US. Around here, the cable company was actually lowering upstream speeds for some users. They raised their downstream though. This was an attempt to encourage them to upgrade to DOCSIS 3.1, I think. Once enough users upgrade they can start doing fancier stuff, like allocating more upstream channels. Maybe by 2030.

Upstream on cable like that will be sub 50 Mbps because they're very intentionally only using a small number of rf channels on the coax for upstream. This is why in my opinion docsis3 is putting lipstick on a pig and is only a short term stop gap solution. The better end state is proper SM fiber to each house.

Yep, even DOCSIS 3.1 modems don't do much for upstream, since generally it remains at 3.0. They have to do physical upgrades, such as amplifier swaps, to actually add more channels and make DOCSIS 3.1 OFDM upstream possible. That seems like years of work, and won't happen any time soon.

That's cronyism for you.

We have a couple fiber companies in our town (USA), and we have been shopping for internet. Currently we have fiber from AT&T.

One is over a mile down the road and they want $90,000 to connect and then $1750 a month for 1gbps. Another company is two blocks away, they will only charge $8,000 to connect and it’s $1,550 a month for the same speed.

Problem is there aren’t many other companies in the area that want service, so we would end up paying all the buildout costs. The other business are small retail, restaurants and shops that don’t need more than coax (150/20 or whatever for $79/month) from Spectrum.

I'm in New Zealand, a small island nation. I have 1gbps for around $70 USD per month. The install was free. I get a VoIP phone and Prime video subscription included.

Most of my devices are wireless so I really don't take advantage of the speed.

Which ISP are you with?

2 Degree's. They bundled it all with my cell plan.

I hit ~400mbps on my pixel 4a's wifi with 802.11ac in a moderately congested environment. With WiFi 6 I would probably saturate

I pay about 17 USD for 1 Gbps in Europe.

Is this DIA service or an ISP?

I'm sure the fact that Germany has a 566% larger population density and is 96.19% smaller than the United States has a lot to do with a countries ability to bring 25Gbps internet to its citizens.

>I'm sure the fact that Germany

Sorry to disappoint you, but it not Sweden or Germany but Switzerland...Germany has bad internet too, but hey that mistake happens even to the best ones (like Bush Junior)

Not a disappointment at all. I see now that it is indeed Switzerland. The mention of the Germany based IX upgrading to 25Gbps towards the end of the article threw me off.

But Switzerland has a 504% larger population density and 99.56% smaller land area as compared to the United States so my point still stands.

Is the density higher then New-York?

https://broadbandnow.com/New-York

>26.1% of New Yorkers have access to 1 gigabit broadband.

Internet in Germany is really bad and expensive compared to Switzerland and even to the rest of Europe. In Germany fiber to the home is maybe 1-2% of all internet connections. Most fiber providers in Germany offer only artificial asymmetric connections and are very expensive.

Solving the last mile facilities based isp problem in detached houses in the USA and Canada is a hard one. In most places with aerial Telecom and power infrastructure, you will have three things coming to a house. Electricity, obviously. And then whatever is the local phone company, and the local incumbent cable TV company.

If you're unlucky enough to live in a place where the phone company is operating old degraded DSL on copper phone lines, and doesn't care to overbuild it with single mode fiber for GPON, and the cable TV company is also something similarly large and slightly evil like Comcast, you're almost out of luck. This is a political and regulatory problem that allows the local franchise agreement for the phone company and cable tv company to be renewed in perpetuity without demanding solid metrics for improvement of service.

If you are very fortunate, there will be an entirely single mode fiber-based third-party provider which competes with the previous two mentioned things. Doing what's called and "overbuild" as a new entrant for this is very capital intensive and requires a lot of physical outside plant cabling work and infrastructure at layer 1 in the OSI model. It also requires appropriate cooperation from whichever local entity owns and controls the wood utility poles (again, a political problem).

This is indeed great, but keep in mind that this story is about one of the well-known greatest: Init7.

They're well-known as a for-geeks-by-geeks service. I was massively impressed by them when I lived in CH.

The primary reason the cost for init7 is so low is that for example in Zürich city the fiber network is owned by the city (like water pipes and power lines) and paid for by the tax payer. Any provider can provide their service over it.

Places outside the city that don't have these kinds of fiber connections aren't this lucky.

Yes and no. Local ISPs in the U.S. are also laying/stringing their own fiber and providing cheap (cheaper than shown here) service. They are starting to move into 10/25 gigabit speeds too, as the costly part (running the fiber) doesn't do much to limit the speed, you just have to update the equipment at each end.

There are a number of county sized public utilities districts in Washington state which are doing essentially the same thing. These particular ones are also the last mile electrical grid operator connected to some of the big hydroelectric dams in the region.

> paid for by the tax payer.

Actually the Zurich build out was only financed by the tax payer to be done by the electric company and Swisscom. Internet providers must each have equal options to lease the last mile from whoever built it out in a given city or town; it isn't free.

1Gb/s in Romania is 10$/month

I feel the pain. I pay $80/mo for 200Mb/s down.

But, still, installing fiber is just expensive.

Much more expensive than almost anything else, and requires skill and experience to terminate the connections, and that's even overlooking any regulatory hurdles of digging along the right-of-way, private and public land ownership issues, the costs of the equipment, and the fragility of the fiber itself once installed.

None of those costs really scale. In fact, they increase linearly as the pool of available installers decreases, and that's not even talking about the cost of connecting that fiber to anything on the other end.

I think I priced it recently and it was $499/mo (+ hidden upcharges) for 1 gig. their lowest price was $65/month for 35/5

I left comcast for at&t fiber, paying $60/month for 1year. If I learned anything since then is to make sure you attempt to leave and they will start calling you with better offers..all of a sudden lol

If you do not live in an area covered by them (which is the majority), it is the same as you say. Situation is not perfect here, either.

I just cannot imagine any use case for 25Gbit down/up.

I have 2Gbit down/up but for a long while I didn’t bother fixing my cables that limited the speed to 100Mbit because I just couldn’t see the point.

The only thing that changed now is that my Steam downloads complete much faster than I expect.

I have a Unifi UDMP, which while I think is probably the best prosumer option right now, falls way short due to a PPPoE problem that limits fiber connections to under 500mb/s, and the lack of load balancing across multiple link networks (which I totally admit, is a first world problem).

I am thinking about building my own, but then comes the maintaince of all of the hardware / software. For me that would be fine, but for my family, it would be a total PITA to manage, and any downtime would be horrible.

So, stuck with UDMP for the moment, until they either fix their problems, or alternatively I decide to bite the bullet and build this on top of linux.

Odroid N2?

Need a lot more power then that to handle 2x 1GB/s connections.

What?

The UDM Pro[0] has a quad-core Cortex A57 CPU running at 1.7GHz, according to the spec sheet I'm looking at right now. It has 4GB of RAM.

The ODROID-N2+[1] has a quad-core Cortex A73 CPU running at 2.4GHz. It also has 4GB of RAM.

The UDM Pro is like half the performance of the ODROID-N2+, accounting for architecture and frequency differences, so... I'm not sure what you're getting at?

The ODROID should be more than enough to handle a gigabit connection if the UDM Pro is even halfway capable of it, from a power perspective. Connecting more ethernet ports to it without an exposed PCIe connector is going to be clunky, but that's not the issue you pointed out.

[0]: https://dl.ubnt.com/ds/udm-pro

[1]: https://www.hardkernel.com/shop/odroid-n2-with-4gbyte-ram-2/

There is far more power in hardware than just the instruction set times the GHZ. Part of the UDM's problem is that there is evidently no offload for needed for PPPoE.

It should also be self-evident, but I also need more the one interface to route traffic to two seperate networks, ignoring that the requirement is actually to load balance multiple systems to a internal 10gb/s network.

The USB is nowhere near what you need to run a 10gb/s or even a 2.5GB/e (based on my experience with other ARM hardware).

No, you really don't. Though the problem in this case is the N2 only has 1 GbE port and adding more through USB, even 3.0, is notoriously bad (regardless of the platform.)

(source: I benchmarked my N2+. It'll route and NAT 1 GBit quite leisurely.)

Can you link or elaborate more on this PPPoE problem? i have a UDMP on the way, and I don't think I'll be affected by it with my simple network design, but also not sure.

It's determined by your WAN provider. Centurylink and google, (AFAIK) leverage PPPoE connections, and the network traffic there seems to be limited around 600mb/s.

I replaced the connection with a cable drop, which I get 1.2gb/s down on.

PPPoE de-encapuslation is likely not hardware accelerated (or can't be combined with other hardware accelerated packet processing) or if that platform is more PC like than I thought, it may be that the PPPoE de-encapsulation is single threaded either as a missed software feature or because the nic can't separate it into multiple queues.

PPPoE is one of the worst network protocols ever, and there's no reason it should have been implemented on fiber. I don't really understand why it was implemented on DSL either; maybe some bizarro way to try to prevent theft of service?

> falls way short due to a PPPoE problem that limits fiber connections to under 500mb/s, and the lack of load balancing across multiple link networks (which I totally admit, is a first world problem).

how are you achieving load balancing? using LACP? not having working LACP seems kind of unacceptable in my opinion for suchs an expensive device.

Also, older enterprise grade routing hardware can be had for very cheap.

It has a failover over the two WAN ports, but no load balancing at all across them.

This is not related to the article, but to networking in general: I just discovered Brocade switches and how cheap they can be for what you get.

https://forums.servethehome.com/index.php?threads/brocade-ic...

Between wanting to dabble with a 10G file server and thinking of hosting 40-60 person LANs, this is incredible.

The most interesting part for me came at the end: It runs router7+gokrazy, a self-written pure-Go userland instead of the traditional GNU userland.

At some point the idea of "use more case fans, then all of them run at lower RPM" simply turns on you and instead produce more noise. For the low wattages of this build a single case fan would've been sufficient to keep PSU and CPU fan at silent levels.

I have a few more watts in my PC case, but not a single case fan, and the machine still hovers at a comfortably quiet level at 1 meter's distance. No temperature issues.

Yeah, I’ll probably turn off some fans to see how the temperatures react. That’s for some time in the future, though… :)

Great :) I’ve been wondering what equipment to get when I upgrade to the 25 gbit connection too. Maybe a custom build is the way to go after all!

I really respect this build, the only thing that I don’t like is the power consumption at idle of ~50 Watt. I understand that this is probably due to the used NICs but still.

It might almost be interesting to setup a second router (a Pi4 might do) for regular casual internet usage + VRRP and only turn on this beast when more bandwidth is required.

The 25Gbit machine would be the VRRP master and the Pi4 the slave.

Not sure why you buy at fs.com when you have online stores in CH that are selling much cheaper:

https://www.microspot.ch/de/computer-gaming/pc-komponenten/s...

To get a grip around the numbers, i quote from George Neville-Neil's Talk at BSDCan '15 "Measure Twice, Code Once" [1]:

- 10 Gbps is 14.8 million 64 byte packets per second - 67.5 ns per packet or 200 cycles at 3 GHz - Cache miss is 32 ns

[1] https://www.youtube.com/watch?v=LE4wMsP7zeA

You're not running a continuous stream of 64-byte packets in a home or SME setup. Also, assuming a 1:1 mapping to packet processing is a false dichotomy these days, NICs are doing an unbelievable amount of preprocessing, particularly grouping related packets together.

No, of course not. A good starting point for real world performance benchmarking could be e.g. IMIX [1].

The example above represents the solely theroretical worst case as a means to establish a baseline for performance benchmarking.

Anyway, if you are referring to HW offloading capabilities of "modern" NIC's, using techniques like LRO would break the "end-to-end"-principle of a router.

[1] https://en.wikipedia.org/wiki/Internet_Mix

Good post, bring me back old memories where I used to setup my own PCs back in high school. I think the graphics card and GPU are really not required for a router. Few years ago there was a Linux project called zebra router which was deprecated in favor of https://www.quagga.net/

@secure: I'm a fan and love what you did with router7, gokrazy and your homelab. Note that this might be seen as "How I replaced my off-the-shelf 600CHF router with a 1700CHF custom PC router".

I hope you'll update this once Fiber7 updates their POP to show measured performance :-)

I have a very similar setup using a desktop as a router (opnsense, 2 WAN, fibre/switch) and it has a single, yet fatal flaw.

The power button. When you lose power, and the power comes back on, your router does not. If you are away, you must physically turn it back on.

Most (all?) PCs should have a BIOS option to always restore power after loss.

You can usually change this in the BIOS (or equivalent). Look for "Power on after AC loss" or similar.

I describe the relevant UEFI option in https://michael.stapelberg.ch/posts/2021-07-10-linux-25gbit-... :)

I wish P4 fpga boards weren't so expensive for hobbyist. You can do fun stuff with them and create a custom network pipeline. It's totally overkill for a home network (they are used to create datacenter fabric) but it's nice and shiny :)

Without PPS measurements its hard to say how good this is. 1.488MPPS is needed to saturate a gigabit connection and that usually needs some tricks like poll mode drives/dpdk/etc on commodity hardware.

I’d strongly recommend people take a look at the supermicro (or your vendor of choice) Xeon-d or AMD embedded 3000 series.

Lower power draw, very quiet and more than powerful enough to push 25gbit with cpu left over for VMs.

Damn, $70 / month for 25Gbps symmetric; meanwhile, we're paying $130 / month for 1Gbps (400Mbps really) down / 40Mbps up in Comcast monopoly-land in PNW.

why no just buy a good nvidia/mellanox nic and route directly on it? Modern NIC's are amazing and you can offload everything if you like.

but it is only offloaded to some ARM cores and not done in hardware? can they really route and NAT 25gbit? and the manuals for connectx-5 or nvidia bluefield don't even mention ipv6?

The article leaves out the actual network packet processing. Is this being done by the kernel or is dpdk being used?

It’s done by the Linux kernel with default settings.

This build would be simpler, better and probably cheaper overall with a server grade motherboard and ECC RAM

One day init7 is going to poach you :)

Reply on news.ycombinator.com