My network home setup – v4.0 | Hacker News Reader

My network home setup – v4.0

giuliomagnifico.blog

>Wiring

A word of warning, it must be said that you shouldn't have a "normal" data cable in the same conduit as mains.

With CAT 6 cable you won't have transmission/interference problems, but still it is not allowed by code, unless the network cable is of the type insulated up to 400V, marked with "CEI-UNEL 36762 C-4 (U0=400V)", see (italian):

https://fibra.click/cavi/#coesistenza-con-cavi-in-tensione

https://www.cavel.it/it/supporto-tecnico/certificazioni/coes...

Yes, I know and you're right but it's only a short path (about 2 meters), and it's the only way I found to get through the cable from a room to another. Anyway I haven't terminated the cable with the wall jack, the cable is going out of the wall "intact", this should be a bit safer.

Sounds like a good excuse to run a bit of fiber

Sure, and as said you won't likely have any issue, and maybe - without knowing - you actually used a U0=400V cable, the norm is 2010 or so if I recall correctly, so I believe that most Cat 6 cables in commerce are nowadays certified for that use.

Just checked, and I see only EIA/TIA 568B.2 ISO/IEC 11801 EN501

You might want to check if that out of code solution could invalidate your homeowners insurance policy. It sure can in the US.

Especially now that it's publicly available on the internet. And yes, the fire investigators for home insurance DO check things like that.

I've seen this claim a lot, but what is the actual legal basis for it? Contractual language in specific policies? Assorted state laws? Homeowners insurance generally covers negligence by the owner (otherwise you'd be out of luck for many types of events). A homeowner modifying wiring in a way that causes a problem would seem to fall squarely into this category.

Legal basis? Ask a lawyer, but I suspect it's buried in contract law.

Homeowners insurance generally covers negligence by the owner

Does it?

I just know several people who have done their own 'renovations' that have had insurance claims turned down (even after arbitration) because they didn't have a licenses contractor do the work. People who insisted they knew better than anyone else. Typical.

So rewire at your own risk, I guess.

If Italy is anything like Spain nobody gives a crap about building code stuff.

When I moved into my apartment it had just been "certified" by an electrician which took a week. There were outlets without covers on them. Exposed live stripped wires hanging in the hallway. Ground wire to the breaker box but not actually connected to the rest of the house. Exposed terminal blocks hanging everywhere. I doubt this "professional" even bothered to visit the place and just cashed a royal fee to sign the paperwork.

It's a total joke. If this crap gets "certified" then a DC cable beside an AC one In a conduit is really no issue :)

Please don't call certification a joke and diminish its value.

If you see clearly illegal things, report them. The person doing the certification can have their license revoked.

Things aren't always ideal but please don't turn this into a laughing matter.

Maybe he did report them and maybe their license wasn't revoked. He's only repeating a joke that the electrician and the certification committee told him. I wouldn't blame the victim, here.

I know HN frowns upon this but location really matters, and op's response is valid and non-laughing in many places. "Certification matters" cannot be resolved as a statement without location / situation.

If I reported such a situation in Canada, it may or may not have satisfactory official resolution before heat death of the universe. One can hope.

If I reported such a situation where I was born, at best nothing would happen ; more likely I would get laughed at. (At worst, electrician and their 3 buddies would teach me a valuable lesson about how things work around these here parts).

Very much depends on location within Canada. BC has an open policy of allowing suites not to code for example

Can you please elaborate? Sadly your linked document is in Italian, which poses a language barrier for most of the community here.

Two problems - mains lines could come in contact with the data lines which would then transmit power to things connected to them (or burn up). Fiber won't do this because it doesn't transmit.

And the second is that mains lines are AC and could introduce noise into the wired lines - again, fiber isn't susceptible to this.

Until that norm (as said I believe around 2010 or so) you could NOT mix low voltage (and signal) cables with mains (220V-240V AC usually) within a same conduit.

The new norm allows this mixing as long as the low voltage cables are certified as having insulation for 400 V.

Still you cannot strip the cable (i.e. you cannot put a terminator/receptacle) in the same box as mains.

The code is mainly about electrical safety, it doesn't consider the possibility of interference, that is "your" problem (but shielded cables give no problems in practice).

The National Electric Code in the US has similar provisions:

> 300.3

> (C) Conductors of Different Systems.

> (1) 600 Volts, Nominal, or Less. Conductors of ac and dc

> circuits, rated 600 volts, nominal, or less, shall be permitted

> to occupy the same equipment wiring enclosure, cable, or

> raceway. All conductors shall have an insulation rating

> equal to at least the maximum circuit voltage applied to any

> conductor within the enclosure, cable, or raceway.

Basically idea is to prevent a low/less voltage cable from potentially being energized by a higher voltage cable. It would suck to strip the ends off your CAT6 and discover it's been energized to 240v.

But keep in mind many times electrical conduit is not used as a bona fide conduit, but rather more of a physical convenience (note how the bit you quoted is talking about conductors not cables. So if you bury a length of PVC pipe (grey or white) with a cat6 network cable plus a UF power cable (required for the wet location, regardless of the pipe), you're fine.

Although residential in the US rarely (never?) goes above 170 volts (peak), which is more forgiving than the ~325V of most of the world.

I believe that a grey non-metallic conduit piece would meet the definition of a raceway in article 100 (to which the insulation rating rules would apply). A white PVC pipe is not “expressly designed” for that use (and would therefore clearly not be a raceway).

Article 100 defines a raceway as “An enclosed channel designed expressly for holding wires, cables, or bus bars, with additional functions as permitted in this Code”

At least in Italy, that would apply as well.

I.e. you can have in same conduit a "protected" (double insulation) mains cable (the type is called FG16 now, it was FG7) and a low voltage/signal cable.

The idea is that the FG cable in itself, having the external insulation besides the single wire insulation and being suitable to "unprotected" installation can coexist with signal cables, the conduit in this case is only an added mechanical protection.

The type of certified 400 V insulated cables I mentioned earlier is instead allowed to coexist with "normal" single wires (this is the normal way electricity is distributed in buildings) inside a conduit.

The issue using FG cables (besides the fact that it is way harder to be inserted in small conduits) might be that it would be suitable to power (say) a mains receptacle (live+neutral+earth) but wouldn't be suitable for (still say) a diverter or a reverse switch due to the colour coding of the wires.

A cable is at least one wire which is a conductor, they use the term conductor because the standard applies to more than wires and cables.

Claiming a conduit is just a convenience doesn't change anything.

Where did you get the 170/325 numbers? In residential applications it's 220-240v for most of the world. North America uses split phase for most outlets but major appliances still run on 220-240v.

I think it's going to come down to the AHJ, or your own judgement if there is no inspection. The way I've seen it explained on electrician forums is that you're not using the piece of PVC for its properties as conduit under the electric code (like say outer damage protection/containment for running single THHNs), so it isn't considered a conduit. But it would obviously be wrong to apply this argument to things like conduit fill or abrasion protection - regardless whether the pipe was manufactured intended for electrical conduit or not.

While they call mains voltage 120V, that's actually RMS voltage and not peak voltage. The peak is 170V, which I think is a better gauge for thinking about insulation and safety (by the time power is being resistively dissipated, you've already lost). Residential "240V" circuits use opposing legs of a split phase, so nothing is more than 170V from ground, which is what matters for insulation and most failure modes. You're not going to get any more of a shock from a "240V" residential circuit than a "120V" one unless you manage to touch both ungrounded conductors at the same time - it's the arc flash risk that gets worse.

Although now that I'm really thinking about it, maybe mixing insulation types is not foolproof at the extremes. If there is a wire with 170V inside sitting in free space, the worst case assumption is that it can have 170V on the outside of its insulation as well (due to the parasitic resistance and capacitance across the insulation, and ignoring the parasitics to ground. If this sounds strange to you, think about how those non-contact voltage testers can work). So if you put a grounded conductor with a low insulation rating right next to it, that insulation could have voltage across it higher than its own rating (depending on its parasitic R/C), which may cause it to break down over time. The possibility is likely moot with real world values, but still.

You can with fiber and it is allowed. I used these special plates[1] in my setup[2] that are meant to be run in the same conduit as power. Switzerland is quite strict with electrical codes so I was surprised when I found out I could do this.

[1] https://www.youtube.com/watch?v=ARSpp4B9-X4

[2] https://sschueller.github.io/posts/wiring-a-home-with-fiber/

Glass and plastic don't conduct electricity.

For glass, that's true in normal temperatures/voltages, but when it gets hot/melts it starts being a conductor instead. Same if the voltage is high enough, see https://en.wikipedia.org/wiki/Dielectric_strength#Break_down... for a table and description about that effect.

Yeah, you shouldn’t. But most of the time it works anyway.

But you’re right, I haven’t seen any professional installation, that doesn’t use CAT 7 for a long time.

Some things I realised after going through my OpenWRT and later OPNsense phases:

- complexity is fun to play with during the initial setup, but it sucks long term

- VLANs and inter-VLAN firewalling is needlessly complex, brings endless frustration*, and you shouldn't trust the network to do your auth anyway

- letting a vendor to do something is Actually Good

- dashboards are useless, I can't recall ever using them for anything

So I sold most of my networking gear and replaced it with

- Aruba Instant On fanless PoE switch and a bunch of their APs

- a £100 Topton fanless PC box with VyOS on it, powered with a PoE splitter

- a UPS

No VLANs, simple flat network. Everything internal is either on Tailscale or behind auth. Everything is PoE, things that don't are on PoE splitters, so no power bricks and everything is UPSed. Arubas require zero configuration and are managed through a cloud portal. The router needed to be configured once and required zero intervention for close to two years. It's ridiculously performant, perfectly balances load, and just works.

*: I really have better things to do on a party than debugging firewalling an obscure protocol Airplay uses when my guest can't Airplay from their phone

That sounds like a really nice, simple setup. I have an unfortunate mix of gear from different vendors, but my setup is broadly similar. VyOS on an old SFF box, PoE whenever possible, etc. My physical topology means I need more layers of switches, though, and I do have a single vlan for my work machine. There's no inter-vlan routing there, just internet.

Is there something that puts VyOS in a separate class than openwrt/*sense? I really liked VyOS when I tried it out, but OpenWRT seems like _mostly_ the same thing. A bit less polished, but more likely to run on different vendor's hardware and let me unify the software without shelling out for a bunch of brand name gear.

I tend to agree on the VLAN stuff. I don't feel like I've found a good reason to do that on my home network (yet, at least). Fanless gear is also great.

For me it was the stability. I haven’t tried OpenWRT, but OPNsense was quite troublesome. I don’t do anything exotic, just standard dualstack and some firewalling, but OPNsense at some point stopped being able to get the ipv6 interface up, with no config changes. I also noticed pings regularly spiking for no discernible reason. I tried to debug both for a few days and just gave up.

VyOS is Debian with effectively a single file config, so it’s both simple and rock solid. As a bonus, pings got a couple ms lower on the same hardware.

Huh, I considered BSD to have a good reputation for stability and I believe network stack performance as well. Too bad.

I did very much like the debian base of VyOS. I've had pretty good experiences with OpenWRT. But it's command line configuration isn't quite as polished as VyOS imo. Interesting note about the pings.

IMO the biggest reason why you'd run OpenWRT rather than anything else is the hardware support. OpenWRT has very broad hardware support for a huge number of commercial devices. VyOS and (pf|opn)sense basically only run on x86_64.

Amen to this. I've followed pretty much the exact same path as you (pfSense instead of OPN though) and reached the exact same conclusion.

I have a sizeable networking background, but still absolutely hated having to keep up with this many moving parts. I very much didn't enjoy troubleshooting this setup.

Dumb is good.

Not the way I went on my home network, but still a good write-up. Always like reading and seeing how people solve problems that go beyond "I bought a 42U rack and installed it in my basement."

I'm going to steal the idea of the Raspberry Pi on the phone stand idea, especially when just hacking around with an SBC at my desk.

I would recommend replacing all those USB power adapters with just one or two dedicated USB power adapters. Can recommend the six-port 60W model by Anker that will happily run all those devices you have, and then some.

> I'm going to steal the idea of the Raspberry Pi on the phone stand idea, especially when just hacking around with an SBC at my desk.

Yeah me too! What model of stand is it tho? and how would you keep them attached? Looking at the pictures it seems different from one pi to another.

Not sure, but my wife bought a number of stands that look very similar, made of 2mm metal with some rubber pads to protect the desk and the phone. Will probably put some heat shrink around the metal arms so that they don't inadvertently come in contact with the back side of an exposed SBC.

You can add PoE (Power over Ethernet) to the Pi 4 or Pi 3B+ pretty cheaply (10-15 dollar hat), and avoid the USB power supply altogether. Not strictly necessary, but makes the wiring so much simpler/cleaner as just one single ethernet cable doing power and data, and you can expand into other neat PoE solutions. My Pi cluster is powered by my ethernet switch alone.

It makes wiring a UPS into the system really easy too - just have backup power on the ethernet switch, the downstream Pis are taken care of. I'd love if the Pi 5 just has PoE out of the box personally, I run all my Pi projects this way now.

I am unsure why you are telling me this. I am aware of the RPi PoE HATs, but the OP looks like he has a number of other USB powered devices beyond a single SBC, and it doesn't look like he has a PoE switch. For $30 or thereabouts he can replace all of the USB PSUs which will simplify the cabling and be a little more energy efficient.

My setup is pretty similar in schematic, but not finess of design.

I have a 24 port netgear fanless smart switch as the backbone. I did have a POE version but the fans were too loud. I have a PoE injector now which allows me to power the APs and the phones for the house intercom.

I use pfsense for routing and firewall.

Ubuquity for APs. I have four, one for upstairs, one for down, one in the garden and one in the shed. three are second hand.

I have a VLAN for work, (I can ssh in from the normal vlans, but I can't get out from the work VLAN)

A have a VLAN for CCTV, normal use, servers/services, and one for IoT. Seems to work ok for my needs, but most people don't need what I want on a network.

Very neat - thank you for documenting this, especially the piece about using Avahi to place the HomePods on a different VLAN. This is something I'm planning to do but hadn't looked into yet, so this will save me a lot of effort.

Just out of curiosity, that's the black box in your cabinet balancing on the metal cones?

Going from the earlier instalments (v1/2/3) - I suspect it's the ISP modem.

(And I'm guessing the metal cones are there to lift it off the flat surface for more airflow).

Exactly, the metal cones are 3 unused audiophile spikes. Perfect fit inside the holes of the bottom of the modem.

V1/2/3 are pretty handy for figuring out the other stuff too.

ISP Modem?

Thanks!

>Very neat - thank you for documenting this, especially the piece about using Avahi to place the HomePods on a different VLAN. This is something I'm planning to do but hadn't looked into yet, so this will save me a lot of effort.

Yes, it's very easy if you use Avahi, but it's important that you're using VLANs and not subnets, because I had lots of troubles using a separate subnets for iot devices and the HomePod in the main subnet. You have to add a route on the router and tweaks the firewall. Using vlans instead is easier and faster.

>Just out of curiosity, that's the black box in your cabinet balancing on the metal cones?

Italian ISP modem "unfortunately". If you see the network scheme you can understand better: https://giuliomagnifico.blog/_images/2023/home-network_v4/Re...

Wait, does this work with HomePod minis? My current mDNS works with my network, my issue is the HomePod mini automatically jumps back to the same wifi as my phone.

Yes absolutely, but your iPhone and the HomePod should be on the same/main vlan, not the HomePod on the IoT vlan.

This looks really super interesting!

I'm gonna check out grafana, it looks significantly slicker than Cacti.

I ended up with a significantly more complex home network than I ever expected -

2 48 Port HPE 1820's 1 24 Port PoE HPE 1820's

All of these are linked with 2 1 GBE links in Port Channel

TP-Link Managed Wifi AP's with controller (I wanted roaming support, and PoE support)

Mikrotik HEx Router also linked in Port Channel to one of the core switches (I'd like to get multiple bonds set up, thats the intent, but I've had trouble making it play nice with rSTP - I think its an issue with my MikroTik Config, but its so poorly documented, its hard to say)

For places where I have lots of port needs where I was unable to pull a ton of cable -

3 24 Port HPE 1810's (2 of these connect back to the Core Switches with port-channels) 1 8 Port HPE 1810 (PoE powered)

The 1810/1820's are great, because they do not have cloud management, are fanless (PoE notwithstanding), and are easy to configure (no weird specific CLI to learn/no poorly implemented copy of Cisco IOS UI) via a web interface. Their lack of 10g support is annoying, but also worth the price savings.

From a VLAN perspective, I have six - one for my external netblock (which is just a pass thru from the cable gateway), and another for my internal LAN, plus two additional VLAN's for my home work lab, and another two for 'utility' which is to say, I built them in, but have not found a use for them yet ;-)

There is also a cacti server in a VM, I need to rebuilt it eventually so I have better instrumentation.

I always enjoy reading about these but man that is a lot of work to set up even if maintenance is simple. Ubiquiti has lost trust but to their credit even a simple UDM base (that is not connected to the cloud) can do VLANs with another device running pihole/wireguard works great. You even could run the pihole on device with podman and use their baked in VPN.

I'd like to plug Ubiquiti also. I'm not a networking guy and I just want my network to work. I don't want to worry about it or try to guess am I having problems due to Comcast or my home network setup.

Switching to Ubiquiti, from high-end Asus gear, has been awesome. Everything just works. Networking is now a non-issue, and when my wife tells me the "internet isn't working", I can respond, "it's not my fault!"

That's worth the cost to me.

I heard some horror stories with new ubiquiti gear, but my ERPoE router has been serving me gbit and PoE for AP since 2016 and 0 issues, it even handles WireGuard using some hoops.

While I will say that our Ubiquiti AP seems to work, configuring it was hell. It involved the management app installing its own nginx instance on my laptop, and then it makes a weird differentiation where if something's a guest network it asks what URL you want to redirect them to when they connect, but if it's not a guest network it doesn't have client isolation. (I eventually found the tickbox to enable it on a non-guest network, it had some weird Ubiquiti-specific name). It was just an ongoing series of problems.

I've been meaning to give out different DNS servers via DHCP on the guest network vs the internal but I just can't face trying to configure that thing again.

I'm glad you're happy with yours but replacing mine with Mikrotik kit is super high on my home-network todo list.

My ubiquiti UAP-Lite was great, until it became flakey. I swapped it for an ancient router that supported an old build of ddwrt, and everything wireless started working again.

Retrospectively I think the ubiquiti AP’s flakiness was caused by a firmware update. This Reddit post is ~2 years old:

https://www.reddit.com/r/Ubiquiti/comments/n46siv/whats_the_...

I’ve been meaning to do a hard reset and/or change the firmware, and put the ubiquiti AP back into service, but the old & slow ddwrt router works fine for everything not an Ethernet cable.

Edit: (note to self) https://help.ui.com/hc/en-us/articles/204910124-UniFi-Networ...

Why is the 100 Mbps port an issue on a device that can never do more than a single video stream. Why should the TV manufacturer spend more money on that part?

First because a TV can last 10 years and have a 1000mbps port will be the minimum.

Second because when you send “something” to the TV like 60mpx photos, using a 100mbps port is slower.

Now a TV is also a home hub, not only a Television. And in the next years the 100mbps will be obsolete very fast.

But if the internal storage of the TV (or the processor) can't handle above 100Mb/s it'll never practically matter.

I've seen more devices that have a GB port and can't do anything useful with it than (I suspect) the other way around.

That said, I've never even checked to see what speed my TV connects at.

> But if the internal storage of the TV (or the processor) can't handle above 100Mb/s it'll never practically matter.

UHD Blu-Rays already exceed 100mbit/sec. That is current commercially distributed consumer content that requires gigabit to stream properly over a network.

Any 4K capable smart TV or streaming device should have a gigabit ethernet interface, no questions asked. 1080p devices, sure, they can get away with 100mbit just fine, but 4K devices have no excuse.

The fact that LG still to this day ships OLED TVs with potentially five digit price tags and 100mbit ethernet ports is a level of cheapness that I can not fathom.

And they handle gigabit just fine, you can plug a USB gigabit adapter in to the TV and it works entirely as expected.

Also, there's like 12 people that care. It's an upgrade that will not move the revenue needle literally at all.

I've never dug deep into this, but the normal argument is that it's possible to saturate a 100Mbps link with a single 4K Blu-ray stream. Even if most people will never hit that limit, it would be nice for a top of the line 4K TV to support "normal" (for some media-savvy folks) 4K streams.

But that's not a very compelling argument on its own, since the Ethernet link is just one link in the chain. Having a gigabit port doesn't help much if the TV can't handle decoding video at those bitrates in real time. It's definitely possible that TV manufacturers choose 100Mbps ports because they know the TV can't deal with huge streams for other reasons.

It's an interesting situation for the manufacturers. Even if 99.9% of buyers will never see streams above 100Mbps, and even if that other 0.1% can't effectively use them, it might be worth it to bump the port to gigabit since complaints about 100Mbps ports come up so often in reviews and in online discussions. Maybe throwing in a borderline useless gigabit port would generate enough sales to justify the marginal BOM cost increase.

As an example, my TV has a 100mbps network port - I thought I was being smart, and plugged my TV into a wired network cable. It kept buffering on a large 4k movie, which confused me because it had been working fine over wifi.

Finally realized my wifi was faster than 100mbps, and hence handled the stream fine, but wired couldn't keep up.

> I've never dug deep into this, but the normal argument is that it's possible to saturate a 100Mbps link with a single 4K Blu-ray stream.

Are people really keeping and 123Mbps or 144Mbps (the two >100Mbps options) 4K Blu-ray rips? The largest 100GB triple layer disc can't even hold 2 hours of video at those rates. Realistically you'll max out at 72Mbps or 92Mbps on 4K discs.

https://en.wikipedia.org/wiki/Ultra_HD_Blu-ray#Specification...

For one, it’s dirt cheap to add what’s basically standard everywhere else. These can be expensive consumer devices and I don’t like seeing sacrifices when it’s completely unnecessary to sacrifice speed here. WiFi is also faster, so TVs can handle the speed.

Buffering. The stream may start quicker with gigabit. Especially when fast forwarding through a 4K movie, fast network can speed it up.

Sony's streaming service (Bravia Core) recommends >100Mbps for their highest quality streams:

https://electronics.sony.com/bravia-core

To access highest quality Pure Stream available at 80Mbps you must have a minimum internet speed of 115Mbps over Wi-Fi. Ethernet (wired LAN) connections are limited to 100 Mbps due to the TV’s product specifications. Therefore, to enjoy 80 Mbps with Pure Stream functionality, you need to connect to the Internet via Wi-Fi (wireless LAN) that supports minimum IEEE 802.11 n/ac.

2 things come to mind here:

1) I don't trust devices to respect VLANs. I trust the switches to respect VLANs, but not devices. When the VLAN-tagged traffic hits WiFi the VLAN is lost. When it's received at the AP the AP can choose to tag it again before entering the switch. I think I'd still do multiple SSID's + VLAN's so wifi clients intended for different VLANs are not communicating on the same "virtual AP"? I worry my Google IOT devices could be in promiscuous mode looking at everything. Multiple SSID's would separate them from other devices by encryption.

2) I've read a couple articles saying rate-limiting IOT and Guest networks results in more service interruption than one would expect. Simply prioritizing the main network traffic over Guest & IOT is a better setup. How do we do this in OpenWRT?

RE: 1, you can push wifi clients to separate VLANs either by host or per SSID depending on the gear. It's enforced on the AP, clients can't breakout.

By host is rather useless, because you can spoof hostnames and MAC addresses.

1) is safe to trust VLANs, especially for this home stuff… otherwise you will need separated LANs and cables! Overkilled.

2) I’m not rate limiting the IoT devices, I’m monitoring them and they make really few traffic, you can limit a device by MAC address in OpenWrt anyway: https://forum.openwrt.org/t/bandwidth-limit-per-ip-mac/35943

> 1)

This is not Area 51 and a client which doesn't respect VLAN tagging should somehow send packets to a different gateway IP. I don't see a way for a device to know where to send packets if it did break out from VLAN

Anyone who uses Grafana to monitor their home setup, thats +1 from me.

Appreciate the commitment and dedication to detail.

You'll love https://mods.factorio.com/mod/graftorio2

My home network has a few differences that might be interesting:

I run openwrt on some mikrotik switches. I started with a mikrotik rb750 switch, then switched to rb2011 switches (5x 10/100/1000 + 5x 10/100 ports), and now two rb3011uias-rm 10-port gbit switches.

the openwrt rb3011 build comes from https://github.com/adron-s/openwrt-rb3011

I also run openwrt on a turris omnia and a linksys wrt1900acs.

I use raspberry pis for a few things, notably standalone ntp time via a few cheap usb gps dongles. One pi does time exclusively and runs openwrt with a gps hat with pps + a pi ups hat. I like the flirc pi cases - they are cheap, beefy and have great thermals.

Why do you prefer OpenWrt over RouterOS on the Mikrotik switches?

I recently upgraded to a CRS326-24S+2Q+RM, and the experience with RouterOS feels much better compared to OpenWrt. Winbox is super polished, everything is well laid out, and it makes even advanced configuration very easy.

I do run OpenWrt on a few APs, and it works fine for that simple use case, but for anything more advanced, I prefer RouterOS. Sure, it's not open source, and not as extensible to allow you to run a bunch of services on it, but those can run on any other server just as well.

last time i checked, CRS3xx not really supported by openwrt.

My point is that the experience of RouterOS is much better than OpenWrt, so I'm curious why someone would choose to run OpenWrt on Mikrotik switches.

on switches - no point. extra no points because openwrt won't support configuring HW features. On the other side I wanted to run OpenWrt on Mikrotik AP, because my other APs at home run it. ALso there is a much bigger selection of 3rd party packages for OpenWrt than for Mikrotik. I guess it will be main point for many people

Yeah, I feel the same way. My main switch runs RouterOS, but I use OpenWrt on APs for simplicity and flexibility. I recently went all in with RBM11Gs for APs, and OpenWrt is a great fit there, but I'm still considering whether I should go with OpenWrt for my upcoming router replacement, stick with my current pf/OPNsense, or double down with a Mikrotik router and RouterOS.

The package selection on OpenWrt is an appealing factor, but I can also run any of those on a standalone server or RPi.

i am using plain debian for past 20 years or so for my routers/gateways. i tried for a while openwrt on ubiquity whatever router, but it felt kind of "wrong".

Thanks for throwing in yet another choice for me to consider... :)

I feel like a router is best served by a purpose-built OS, that is heavily focused on that task, and restricts the execution of arbitrary software, for obvious reasons. I suppose one could single-handedly customize a general purpose distro for that task, but I'd rather trust a group of dedicated and more talented hackers to do this for me.

That, and I'd rather not manage nftables rules directly. :) Though it would be a great learning experience, so I'll think about it.

>I feel like a router is best served by a purpose-built OS, that is heavily focused on that task, and restricts the execution of arbitrary software, for obvious reasons

well, in this case, unless something changed recently this is definitely not openwrt

RouterOS is a terrible operating system.

Well, that settles it then!

(care to elaborate?)

I didn't like routeros because it would try to connect to strange ip addresses out of the box.

I like that with openwrt, it doesn't do that, and you can configure all kinds of things just like you want. At first I would use the regular releases and install the packages I wanted. As I got more comfortable with it, I would just build it myself.

It's pretty easy:

   git clone https://github.com/openwrt/openwrt
   cd openwrt
   ./scripts/feeds update -a
   ./scripts/feeds install -a
   make menuconfig
   make -j$(nproc)

make menuconfig is where you choose how your system is configured (packages, kernel modules, config settings, etc)

my initial builds were sort of experimental, but it was kind of fun and eventually I learned to customize exactly what I wanted. For example I would use ipv4 only and strictly control the ip addresses of all my machines. (current openwrt doesn't allow it, you have to turn off ipv6 using sysctl). I configure out wifi/bluetooth from some machines that don't or shouldn't use them. I set up privoxy and some machines do updates through the proxy which whitelists what they can get to. I use vlans, and it keeps traffic segregated well. It's nice to put a weird device on a vlan and know it won't go uploading to the cloud, or update itself without your say so.

Just a tip, after

  make menuconfig

run

  make -j kernel_menuconfig

To select some additional options for the kernel, then

  make -j $(($(nproc)+1))

That can be a bit faster nowadays.

Thanks a lot, this is great info!

I had no idea about RouterOS connecting to strange IPs, I'll look into that. Can you link to some research that confirms this, why it's done, and how it can be disabled?

I do like how configurable OpenWrt is, and didn't know it was that easy to make a custom build. I'll probably give that a try the next time I have to set it up. Thanks again.

Sorry, this was a long time ago, and my switches are all running openwrt now.

Thank you for posting, very informative.

Some random comments:

I use a Mikrotik router and I have a dedicated network for devices I don’t want to access my main network. They can only access my MQTT server. RouterOS (mikrotik OS) is a bit terse and comes with its own cli interface. I managed to modify the default setup relatively easily via the UI to create the two networks I needed. In the future I may install openWRT but at the moment the current setup works well.

Another think that I did recently that was quite impactful (performance wise) was to add a Omaha controller to make my two access poins work together to expose one single wifi network. Before I had them working on their own. The performance of the network has increased substantially. I’d prefer not having to buy a piece of hardware to do that but I am glad I did.

I recently bought a ds720 from Synology. I upgraded the RAM to 6gb. So far I use it to dump my personal backup (restic). Also my Reolink cameras dump video via FTP. Because the ds720 runs linux and docker I am planning on consolidating a few services in the ds720 (home assistant, grafana and Pinole).

Oh, I also got a UPS system for the main components that provide Internet access. I can be without power and have Internet for 2 hours a half. One thing I want to do is to get an alert when the power goes down. The UPS exposes that via USB.

Wow! I'm excited to hear someone has the WAX206 running openWRT. I've been trying to get more of my networking gear running that, but since the WAX206 isn't officially supported I hadn't looked into things further.

How much of a pain was it to compile? The post doesn't seem to describe any changes made (understandable, most people probably don't care) but there must be some changes if you had to compile it yourself, right?

It’s easy, you just have to clone the fork with the support and then select the packages and compile it. The fork is this: https://github.com/boretom/openwrt-fork/tree/wax206/flashabl...

If you want I can send you my image.

I will try building it, since I also want to get some experience for making it run on my Mikrotik RB5009. But if I run into issues I may take you up on that offer! :)

Thanks!

Yeah correct, try before is the right way!

Is it working fine to have IOT on a different vlan, lot of IOT use weird protocol ( mdns, multicast etc ... ) that are not friendly with vlan? I know that some people have issues for example with the Chromecast being seperated since it needs internet but also be able to communicate with your phone on a different vlan.

Yeah, avahi will help you out quite a bit there, but I personally pick my IOT devices to where they will not have requirements like that. I'm pretty #nocloud with anything I put in my home, so the majority of IOT devices I have go on the null routed VLAN and are perfectly happy.

My IoT VLAN is one way only (main VLAN can talk to it, and it can talk back BUT it cannot talk to any other VLAN on its own accord). No issues with mDNS or multicast. I redirect all DNS request as well to nextDNS with masquerading. I have probably 30 devices on it? Zero issues with home assistant and HomeKit

IoT VLAN indeed can be annoying. It's getting better as a lot of the more "prosumer" grade routers are supporting it. I use Sonos at home too, which means I had to deploy this into a VM to bridge the VLANs: https://github.com/alsmith/multicast-relay.

There are some funny (?) things that turn up too, like learning the Roku remote iOS app "discovers" devices by opening a TCP connection to every address in parallel on its local /24 (!!!). It sends out and receives mDNS packets that would tell it exactly where they are, but they are ignored by the app.

Assuming you have a linux machine connected to both networks, Avahi can reflect/forward mDNS multicast traffic, so you can have your chromecasts on a separate network and be discoverable by devices on a different one.

Multicast doesn't cross between IP _subnets_ - it doesn't necessarily have to do with VLANs, strictly speaking. But yes, in practice

    VLAN←→subnet

Make sure IGMP is enabled. Devices join IGMP groups to announce they want to receive mDNS

- IGMP snooping

- IGMP proxying (if offered)

Depending on your router you might find helpful options like:

- mDNS reflector

- mDNS repeater

- any mDNS + description of multiple networks (Unifi)

    tcpdump -i <interface> host 224.0.0.251 or port 5353 -A

Like others mentioned, Avahi is solid but the multicast reflection/repeater/relay must run on the device routing between the VLANS in question.

Disclaimer: Deployed and networked thousands of Chromecast at several hotel chains and their wildly variable enterprise networks. Wrote my own mDNS repeater-as-a-packet-rewriter to fine-tune TXT records.

One thing to takeaway is that wired is so much better than wifi.

At home I am just using the ISP router but I have my work laptop,desktop, consoles and TV wired with ethernet and it is amazing compared to wifi. No more dropouts, random ping spikes/lag etc.

Just ISP router with 4 gigabit ports + one Netgear GS108 dumb gigabit switch.

Can someone recommend a budget WiFi access point with long range? I only have LTE as the backhaul, so the fastest speeds are not a requirement.

I bought a EAP610 which I saw recommended on Reddit, but the range seems worse than the ISP modem's (something Huawei) built in WiFi.

for a normal AP, then I'd get a second hand ubiquity LR off ebay.

My advice is tangential but run an ethernet cable. Access points aren’t great at long distance. Setup an AP in the far away room on the other side of the house. It will be far less frustrating.

I bought a Netgear WAX218 a few months back for around $100... but a quick look around shows that either the price has gone up significantly or they're not making them anymore? Well, if you manage to find one for a decent price, I highly recommend it.

I am a big fan of Netgear’s Orbi line. Really I think distance is more of a relative/ location issue and a mesh system that allows you to move the satellite endpoints around to suit your needs is very useful to figure out the optimal situation for a given environment.

https://www.netgear.com/home/wifi/mesh/orbi/

Buy two Asus RT-AX56U, install Merlin on it, then join them using the built-in wired (or wireless) mesh feature to cover large area on the cheap.

Loved this! I have just now started rebuilding the home network, this is great inspiration

The link for a grafana chart full image doesn't work:

https://giuliomagnifico.blog/_images/2023/home-network_v4/Sc... vs https://giuliomagnifico.blog/_images/2023/home-network_v4/Sc...

Fixed, thanks!

What I find interesting and impressive

1) your photography

2) your HN account is ~3 years old, with 33k karma.

Ahah thanks…but I spent lots of time in writing this article =)

I’d highly recommend a Ubiquity Dream Machine Pro if you have any advanced use cases. I’ve got mine VPN bridged to my office router and it’s been convenient to be able to force some devices at home to have all traffic routed over that link.

PlayStation dev kits annoyingly require usage on a whitelisted static IP to activate (every 2 days) and access dev PSN environments. It would have been a huge PITA doing it any other way.

I wonder why noone is talking about the 'bufferbloat' issue, is-it a solved issue now? Can I pick any router to plug into my ISP router?

It’s a solved issue if you’re using the right hardware for your isp connection.

What does this mean? How can I check if I have the right hardware?

Depending on the speed of your ISP connection, the faster is the connection, the powerful should be the router

The name "bufferbloat" is falling out of favor. First off, it has a horrible sound... And it doesn't really give an intuitive sense of what's wrong.

Referring to "latency" or (my favorite) "responsiveness" is better.

And I was encouraged to see this recent ZDNet article that mentioned the "ping rates" of 600-1000 msec, and notes that these would cause videoconferencing or gaming to be unusable.

https://www.zdnet.com/home-and-office/networking/i-tried-del...

And as @giuliomagnifico points out, you can fix it yourself if you get the right router.

I've been waiting for a good time to ask this oddly specific question: why does everyone number VLANs 10, 20, 30, etc. instead of 1, 2, 3?

At least in our case, this allows this space:

172.16.0.1 to 172.16.9.255

To be available for non-VLAN DHCP, static leases, and internal devices. Not sure if that's why others do it this way, but it made sense for us.

Originally, so you could group related VLANs together. e.g. VLAN30 is Marketing, then later you need a second marketing team so they have VLAN31. If you’d had VLAN1, 2, 3, etc, you couldn’t do this.

That everyone does it - even on small home networks - is just convention.

Same reason as assigning larger networks than you need or leaving free spaces between them. You may want to put some things close to each other because they logically go together. But some things that go together don't exist yet, so let's reserve the space.

(Can't speak for everyone of course, but that's why I'd use 10.0.10.0/24, then 10.0.20.0/24, etc. Now "same kind of thing next to it" can have 10.0.11.0/24)

Because VLAN 1 is the default used by lots of vendors, and sometimes also 2, so using 10 and 20 is easy to remember that is a VLAN and you can leave some static IP free also. Also because is not like DHCP addresses that are 1-255 but VLANs are 1-4096 so you can use some easy numbers to remember. For example I’m using VLAN 50 for IoT because the Homebridge server has 192.168.1.5 IP, so IoT is VLAN 50 with 192.168.50.0/24.

Some can argue that using VLAN 1 is also a bit less safe because it’s the default VLAN and attackers usually scan for it like 192.168.1.1 IP for modem/WAN.

On some devices (e.g. CISCO), ID 1 is reserved, so starting at powers of 10 keeps it nice and even and allows for insertions (same logic as line-numbering in BASIC.) I assume 10 seems better than 100 (or even 1000); those just seem crazy high.

Is there a good reason to use VLANs? I’ve never really seen the need for it. It’s good practice to secure all your services even in a home network. And in the end you have access to all the services anyway at some point.

Also why 3 raspberrys, instead of one with a few containers on them? Especially the 4s draw quite some power, just from a power saving perspective I would only run one.

Yes there is (VLANs) and that's why they've been a thing for quite a while. Securing stuff might mean things like firewalls. When you are defining firewall rules you need to define where FROM where TO and WHAT and ACTION. This is what out of the box TCP/IP gives us. It can be rather more complicated but we are discussing VLANs.

You might have PCs, servers, TVs, printers, cameras and more on your network. You might want some of thos to access the internet and some not. Some from the internet and some not. Anyway - policy - what should be able to get from A -> B.

VLANs allow you more flexibility. You can now have lots of different TOs and FROMs. So you can put your security cameras on a VLAN with no access to the internet. You can still access them but they cannot splurge to the wider world.

Three RPis? Perhaps. Depends on the job. I'd probably throw another VM on the fire.

(EDIT - grammar)

Your point is more about subnets and less about VLANs. You can have firewall rules that restrict entire subnets from access the Internet; you don't have to define a rule for each device. VLANs just give you assurance that a device can't just change its subnet to your main one and gain access that way. Realistically, there wouldn't be any IoT device that would do this. But I agree if you can do VLANs, you should do them over basic subnets.

I get that. I know what you can do with VLANs.

But why do you need it in your home? Do you really cut off your printer in a VLAN and do some specific routing/filtering? I know how to do that, but I just don't see the benefit.

If you want to cut off one device from the internet, my solution would be to set a specific DHCP rule to not deliver a gateway/dns. Easy and good enough to cut off a printer from the internet. My home does not need the same network security as a nuclear power plant...

I don't have pictures but I can describe it.

* Broadband 600/60Mb/s with seamless failover to 5G (varying speeds)

* Netgate 6100 router with VPN client, VPN server, site to site VPN configured, traffic shaping to reduce bufferbloat, uplink failover, etc.

* 4 Cisco SG 250-8 switches sprinkled throughout the flat. One acting as my core switch.

* QNap with 2 4TB drives in mirror for backups

* A HDD USB station with a stack of 4TB HDDs for backups. Backups are delivered to qnap at various times and then from time to time I make a complete copy to a drive which is put in a rotation. I keep three full copies of the data at any time and at least one of them is off-site with my family. When I visit my family I take the latest backup and replace the drive that is in their custody.

* a small, passively cooled server with 2TB fast SSD, 128GB ECC RAM, Ryzen 5 CPU, Asrock PRO X570D4U-2L2T. Hosts proxmox where I keep about a dozen VMs for various things, Ubiquiti management panel, NVR, dns filter, development tools, minecraft servers, jump box, etc....

* a 10 year old Thinkpad T440s running always on serving as my emergency server and a development environment.

* 4 Ubiquiti WiFi 6 access points -- before you jump in saying this is overkill, I live in a large flat in a dense urban area with about half a thousand 2.4GHz APs and 50 5GHz ones interfering with my WiFi setup. Most people and even network providers are clueless and set up their devices to max power as if it was going to help them -- it only makes things worse. I have 4 APs with reduced power so that anywhere you are at my flat you are always close to one of APs and you roam between them seamlessly as you move.

* Multiple VLANS and WiFi networks

* a VLAN + WLAN for my family for their regular devices to access the Internet and some defined services within network but otherwise disallowed to contact anything else

* a VLAN + WLAN for IOT, legacy devices, devices I don't trust or devices that only support old protocols and would deteriorate WLAN performance (printers, a chinese projector, etc.) This VLAN does not have Internet access (so that devices can't phone home), don't have access to any other device in the network, don't have access to other networks and can only be reached with defined firewall rules.

* a VLAN + WLAN for my work -- this is dedicated for my work laptop, my phone, my electronics lab (oscilloscope, multimeter, programmable PSU/load, etc.)

* a VLAN + WLAN for guests

* a management VLAN -- any network devices, servers, QNAP etc. are only available through this separated VLAN which has very strict access through a jump box. Also does not have direct internet access so the devices can't phone somewhere else (but I have a proxy for software updates, etc.)

* a service VLAN -- where my services are available internally (for example QNAP interface, apps running in VMs, etc.) Some of them have rules to be accessed from other networks

* a DMZ VLAN -- I expose some services to the world, DMZ serves to provide one more hurdle for any attacker

Thanks for sharing this!

I'm a networking amateur, and one thing I've struggled to figure out is VLANs for wireless devices. It seems like VLANs are managed at switch level, so does that mean that all devices on a particular AP have to share the same VLAN? Or is there a way to segregate devices across multiple VLANs within a single AP?

Enterprise APs support VLAN tagging themselves, so you assign multiple VLANs to the AP uplink in the switch and then tell the AP which SSID belongs to which VLAN.

Yes. I set up VLANs on my Cisco switches. The APs are told what vlans and WLANS are configured through Ubiquiti management panel. The APs are all connected to their assigned ports on the switches and the ports are configured to see all necessary VLANS tagged and one (management) VLAN untagged. The untagged VLAN is how the management application talks to APs.

Eeach of 4 APs serves all 4 WLANs and each WLAN + VLAN are completely separated networks.

The traffic from various WLANS goes directly to their assigned VLANS and never mixes together -- the only way is either through the router or some other service like my proxy.

Gotcha, thanks for the extra details!

Is Aruba Instant On considered an enterprise AP? It is the cheapest and easiest way to do home networking with VLAN that I have found.

If you read my post is what I've done: separated VLANs (3) with a single AP and cable from the router.

>I don't have pictures but I can describe it.

That's very interesting, but how much power does the whole thing consume?

In my case all this setup is 45-50W, I thinks is a good goal.

I don't know how much all of this consumes. The networking itself is pretty power hungry, just the APs probably consume more.

On the other hand there are no fans in my setup except, incredibly, the laptop. But this fan is kicking in extremely rarely and only when I am actually using it, so no problem.

The backup NAS makes a bit of noise but this is happening during night when nobody cares.

I like this setup. Mine is much simpler, but I dig your vibe with the VLANs. I don't have any Internet failover or VPN, and have settled on:

- Regular VLAN: Access to LAN and Internet (I insist on having root on the device for it to go here)

- Guest VLAN: Access to Internet only

- Quarantine/IoT VLAN: Access to LAN only

I don't feel I need any more granularity than that. Of course the primary LAN backbone is 1Gig ethernet, but I have APs every 50 feet or so for phones.

I thought about 10Gig but then I decided almost no device I own can actually make use of it and even if it could, there are better ways to do it. I don't need to have 10Gig just to be able to edit videos/photos if I can easily solve the problem and copy them locally for the duration. Also almost everything uses WiFi and there are only two computers (my macbook pro and gaming PC) that are connected to ethernet.

As to APs, having multiple APs (well configured) and a good router (well configured) has much bigger impact on the quality of user experience than the actual throughput of the broadband itself.

As a homelabber myself (enterprise networking + servers) there are quite a few things to consider before jumping ahead with such a setup. It can be rewarding but you'll need to commit to it and be prepared to troubleshoot - you're basically a small business IT shop at this point. Having some network/IT background is obviously helpful.

Keep in mind that the power consumption of all the equipment is quite substantial and must be taken into account before starting. Also as your setup becomes more complex backups, redundancy, and security must all be considered - it's easy to run your network dead in the water if you aren't prepared for it, and unlike a single home router you can't just simply reboot and reset if everything relies on the network. For instance assume that all your machines rely on your NFS server to access files - if that server goes down, how quickly can you replace it? If the RADIUS server goes down and your devices can't authenticate across your switches and APs, do you have a fallback method of access?

Finally unless your family knows how to maintain the system as well, you'll be the sole IT contact and will have to do quite a bit of support especially at the start. You'll need a plan of how to remotely manage everything if you're say on vacation since things like to crop up then.

> As a homelabber myself (enterprise networking + servers) there are quite a few things to consider before jumping ahead with such a setup

Well. I have over quarter of century of experience in IT, as a sysadmin, developer, electronics engineer and tech lead. It helps. I would never suggest anybody to do this just to have a nice WiFi at home...

> Finally unless your family knows how to maintain the system as well, you'll be the sole IT contact and will have to do quite a bit of support especially at the start. You'll need a plan of how to remotely manage everything if you're say on vacation since things like to crop up then.

Yep. I have VPN I can use to manage the network. All devices can be rebooted remotely.

I also have some backups -- the 5G router can be disconnected from the setup and used standalone and I have instructed my wife how to do this. Most of the files are synchronised to a cloud service where she can connect in need.

The passwords to everything are stored in tamper evident envelopes (and a paper books with a log in my own handwriting).

As to power consumption this probably is the weakest point of all of this. Yes, a lot of devices equals a lot of power, but my devices are extra power hungry. Although I tried to avoid unnecessary electricity waste (if only to keep it fanless) I never compromised quality for it. For example, I went out of my way to not buy an actual server even though there is a plenty of used servers that I would be perfectly happy with. Instead I built my own based on one of a kind motherboard that supports a consumer CPU and ECC RAM and uses relatively little power.

Hah from reading your original post I already knew you were good. My comment was really meant for those interested in these setups (I get asked about this quite often) without realizing the time and effort needed to maintain it. This can be a real rabbit hole as I started with an Edgerouter and Unifi AP and eventually worked my way up.

I really like your idea of having a separate router that can be used standalone if the main system fails, and might actually consider adopting that for my family as it would be very useful if I'm not available. Currently I'm looking into a virtual HA Opnsense setup on two servers to maintain routing if one fails and cannot restart for whatever reason.

We take this router with us on trips. It is nice to have your own fast, mobile Internet with you (no transfer or bandwidth limits). And when it does not serve as backup Internet it has site-to-site VPN to our home network.

I recommend anyone separate VLAN for your work at home environment. The company might spy but far more importantly, the risk of viral infections and hacks is so dramatically higher in a company than you alone at home with your family.

Yep, that's what I have.

One large bank I worked for was very surprised and practically enraged when they figured out I work on a VM and they don't actually control the device I am sitting on. It all started because they decided I am obliged to "provide for basic security" and install an antivirus. I told them there is absolutely no need for me to install an antivirus on this machine. This machine has only ever been used to connect to their network and I have neither installed anything or even visited any website from it. Moreover, it is snapshotted and restored from a snapshot every single day. It is fun to sometimes battle those mindless corporate drones.

What cooling is on the passive ryzen 5?

It is Ryzen 5 Pro 5750G. It is a unique CPU that supports ECC, has 8 cores 3.8GHz up to 4.6GHz (boost) but only 65W TDP. There is lots of options available but in the end I decided to make my own.

I use a separate router and old phone without a sim card to manage my IOT devices, got sick of Amazon continually scanning my network and adding my printers without asking.

I know it happens but I hate that these devices probe my networks and report on what they find. Is there anyway to stop this discovery?

Hardcode IPs and disable broadcast traffic. But really VLANs is the answer.

Thank you, I wonder how many IOT devices support entering an IP address directly.

> Is there anyway to stop this discovery?

The correct way is to create VLANs. Then use the router's firewall to prevent devices in the IOT network from reaching into your other networks. Not all consumer network hardware supports VLANs though.

Thanks I meant the discovery on the IOT LAN or VLAN.I don't need Amazon knowing that I have a Tesla charger

Gotcha. You can never tell how an IOT devices is scanning your network. It could be passively listening for broadcast messages, or it could be actively scanning all the private subnets.

So, you probably need an access point that can do "client isolation" or "layer 2 isolation". This would prevent clients on the same wireless SSID from talking to each other.

For example, looks like the Ubiquiti access points can do it. https://evanmccann.net/blog/2021/11/unifi-advanced-wi-fi-set...

Thanks for the great info.

My separate router allows me enable 2.4 G which many IOT devices need but keep my main router at 5 G only.

That's a good idea when you're just working with what you might have on hand. But if you're buying something, consider going a step above consumer network gear. There you'll find wireless access points that let you configure multiple wireless SSIDs on mixed or isolated radios...all at the same time.

I love that you modified a piece of furniture. I plan to do something similar with a rolltop desk.

Are there any server rack mounted patch panels that let you choose to use a certain network drop for POTS or for Ethernet?

I’ve seen similar patch panels for structured wiring, but not for server racks.

With the 19" front rack mount I have never seen one.

Get a 19" keystone panel and then you can do whatever you want. There are keystones available for Ethernet, coax, rj11 POTS, hdmi, fiber, basically anything.

What about the tv?

It’s on your main net, but guests sometimes would like to connect for for example some streaming.

Also, Plex should be on the tv network, but accessible (Uni directional) from the main net

At the moment I don’t want the guests be able to stream on my tv, I don’t use Plex. I use AirPlay to send video to my tv.

I'm looking for the fastest possible switch for 6-8 cat6 lines. Anyone know of any affordable options (preferably in the UK)?

Very curious, what if you had a 10gbe symmetric connection from your ISP? How would you modify your deployment?

...well I think it takes a long time before we will have 10Gbe in Italy (we still don't have 5Gbe), anyway I'll use only another router and switch, with 10gb ports, but the issue in this case will always be the wifi antenna of the (i)Devices that are still below 1Gbps, so the AP will not need a swap at the moment.

There is at least one provider offering 10Gbe in Italy (in download - 2.5 in up): https://www.dimensione.com/portale/fibra-internet-10-giga.ph.... There may be more that I don't know of.

Yes but there’s no real coverage, just few houses in the center of Milano or Torino. Where I live isn’t available.

Yes, even if I live in Milan, my house is not covered. Though it's nice to see things are starting to move in the right direction.

how does someone learn the basics of "home lab" or small-scale server setup, particularly networking?

I'm pretty familiar with managing compute & storage, but the networking is largely a mystery to me. I've read a bunch of CompTIA study materials but it was all very abstract

Tbh a lot of it can be as simple as:

- get computers. laptops, desktops, raspberry pis, custom-built ("whitebox") servers, old dell poweredges you got off ebay, etc etc. Install linux on them.

- plug servers into switches, switches into switches, and eventually into your router. Don't create cycles in your tree (unless you know your router/switches support it (STP), and unless you paid $1k for your switch, it doesn't support it)

- Figure out your router config to assign them static/reserved DHCP IP addresses so they always get the same IP.

- put those IPs in your hosts file. (optionally, set up a DNS server.)

- ssh-copy-id your ssh key to all servers

Now you have a bunch of machines you can ssh to. Which imo is the most basic definition of a homelab.

Lots of people get super creative and use fancy routers and switches and enterprise gear and do complicated networking and etc etc etc but all that stuff is just good fun and not necessary.

Best thing I think is "do it", because when you need to fix an issue you learn new stuff, I have never done dedicated studies, also because each system has its own particularities, so you can learn the basic but then the names and operations may change a bit from one to another brand.

I think you would benefit from an "Introduction to Computer Networks" type class

It will teach you what a switch and a router do, the difference between LANs and WANs, what DHCP and DNS do. The different ISO/OSI layers involve, TCP vs UDP.

Then you'll be able to setup a home network without issues, because you'll know the different moving pieces and how they fit together.

This is a textbook that's used in such classes

https://intronetworks.cs.luc.edu/current2/html/

From the syllabus, this Coursera class looks OK:

https://www.coursera.org/learn/computer-networking

Anyone have a MOOC or other course on this topic they've taken and would reccomend?

In the context of the linked article, the easiest starting point would be to get a managed switch like the Netgear GS308T in the article, and then feed the data into grafana for pretty graphs. From there you can start branching into more complex topics like vlans, wifi, etc

Download cisco packet tracer (you need to sign up for a learning account):

* https://www.netacad.com/courses/packet-tracer

It is network simulation software that simulates down to the hardware level and will let you setup networks and see how they work as individual packets move through.

nice setup!

I spent weeks searching for a 10G switch that can support the IEEE 1588v2 PTP Transparent Clock (TC) mode but couldn't find anyone that can fit into my budget ($1-2k USD new or <$1k for used). Anyone has such switch to recommend?

I'll just say one thing regarding my own home network setup: go IPv6 only. Ditch IPv4, except for the necessary evil that is NAT64/DNS64. I refuse to network any device that does not support IPv6, and I refuse to use any app that chooses not to use the IPv6 addresses present.

Why do you do this? Principle, or does it have an actual advantage?

Mostly principle. The internet is designed for end-to-end connectivity; let's strive for a more decentralized internet by giving big cloud and residential users equal access by removing NAT.

As for actual advantage, I can think of reduced configuration burden since you don't have to maintain two sets of firewall configs for dual-stack hosts. It's a small advantage only.

On the other hand, I'll be honest with you, there are disadvantages. As recently as 2021, people are still discovering problems on IPv6-only networks that necessitate writing new RFCs to mandate new behavior. Yes I'm talking about https://www.rfc-editor.org/rfc/rfc9131.html It's because of the low prevalence of IPv6-only networks that changes as fundamental as Neighbor Discovery have to be proposed in this decade.

I think that's a nice framing for the issue! IPv6 adoption is really slow, considering that I've been hearing about the necessity for what seems like two decades now.

Yes, because I want my internal home devices publicly accessible by default.

Seriously, the global addressability of ipv6 is something that people used to using ipv4/NAT tend to forget. I know a bunch of people (well, two) that make a living scanning for IPv6 addresses inside networks that the admins didn't realize were open to the world.

I've been trying to embrace IPv6 as much as possible in my home setup. Something funny I noticed, though, is that ICMP ping times are 30ms higher over IPv6 vs. IPv4 between the same two hosts when the packets go over a wireless WDS link. I have yet to explain it.

Yeah ISPs route IPv4 and IPv6 traffic differently. They are after all independent inter-networks.

I'm talking about my own traffic inside my LAN, though. It's all switched or bridged at layer 2, without any layer 3 routing. I get 30ms vs. 2ms ping times, where the only difference is the IP version.

My current theory is that the WDS link devices' wifi firmware or drivers are doing some sort of packet content based QoS. They see the IPv4 ICMP ping request go by and optimize for return latency. The IPv6 ICMP ping request, on the other hand, doesn't. It's like diesel emissions cheating by having the car detect it's on a dyno...

Addressability != accessability of course.

Stupid ISPs distributing /64 to their home customers.

I love that it looks so rustic.

No IPv6?

My unnecessarily convoluted home setup that takes too much space - ftfy.

Agreed, but it's neat.

Every time I try setting my home network up like that (smart firewall, traffic graphs, etc), I just end up going back to a $30 router/AP.

Had a similarly convoluted network for some years... over time you realize it's just pointless to waste time maintaining and troubleshooting said setup.

Today it's ISP router + separate AP (better coverage). Chinese hackers aren't attacking my network, and if they did, cool, have at it. Basic firewall + NAT + AV covers 99% of use cases, even in a business, with the right configuration. Turns out I don't miss pfSense either.

Makes sense for keeping skills up to date, though, and as a hobby, I can see how one can get into it. Reddit's r/homelab has some crazy builds to check out.

I essentially have a foot in both camps... I like having the control and autonomy of open-source networking hardware but I don't have enough spare time to make it a full-on hobby. Right now my "happy spot" is:

1. An OPNSense firewall between my cable modem and the rest of the network running on a low-power PC Engines APU2. The web-based UI is funky but workable, full SSH access to the box for digging into the internals when needed, online upgrades are a cinch.

2. An 8-port gigabit unmanaged switch that everything hangs off of.

3. A Netgear WAX218 business-grade access point for wifi, running the stock firmware. Web UI is decent and doesn't require any cloud-based management bullshit. For around $100, it works much better than it has any right to, given the prices of mid-range APs and wifi routers these days.

4. A small fleet of Raspberry Pis for miscellaneous tasks.

If I get more into IoT, it shouldn't be much of a hassle to add VLANs and maybe another switch.

That sounds like a good "happy spot" and doesn't veer in hobby territory IMO. More like an interest.

In retrospect, I lied a bit about not missing pfSense (or OPNSense in your case) because truthfully I miss the monitoring, packages, configuration and expandability options. At the same time, I also don't miss them, because 0 headaches and actually better latency is still a plus. Just need to login to that god awful ATT interface to open up a port, but these are 1st world problems... there's always VPNs and cloud VPS to fix that.

Unless you're really into managing a small fleet of devices for basic functionality I'd highly recommend replacing them with a single Intel NUC or similar. I did the same after one too many SD card failures and was very happy with the results - you get a significantly more powerful server for a power footprint about the same as all the horribly inefficient USB power adapters running a bunch of Pis.

I'd sub the ISP router for a £120 topton box with vyos on it, just because it can handle smart queues at line rate. It's really nice when you have exactly the same low ping and jitter regardless of other load on the network, with bandwidth splitting equally, and ISP routers just can't do that in my experience. It just works and requires zero fiddling.

TBH, haven't gone into anything deeper than a ping and jitter benchmarks, so not terribly in depth or long-term besides occasional tests out of curiosity.

ATT fiber 300 up/down provides 4 ms consistent ping to google's closest's datacenter, sometimes at 3 ms, which is of course nuts. Might as well be in my apartment block. Perfectly happy with provided unit, although it's an older one.

Tangential, but have used vyOS some years ago to create a makeshift 10G switch using commodity hardware and an old PC. Routed and switched amazingly fast - the demise was related to what I could guess were broadcast storms.

I'm with you in spirit however. Want and will probably need to switch back to a more customizable router.

I have something relatively similar, a bunch of old datacenter equipment (cheapest way to get 10+ GB!) and some mikrotik, and then I have hardcoded DHCP leases for my IoT shit, and extensive blocking at the firewall for those devices/MAC addresses.

Good enough for me.

Are Fritz!Boxes available in the US? They're built by AVM (a german brand) and are pretty neat if you want something that's secure, supported for a long time and easy to configure. Add some of their wireless repeaters for coverage via mesh networking and you'll have a guest wifi available everywhere and all is well.

I had one of these boxes and found it to be beyond infuriating

I would set up something simple like port-forwarding to a static IP and test that it worked

then I'd come back a few days later to use it and found the router had helpfully changed the IP to another one

and this happened with several different features (IPv6, DHCP, etc)

I replaced it with a much cheaper Mikrotik box and that's worked flawlessly ever since

I would not recommend the Fritzbox to my worst enemy

If you select a host in the network overview, there is an option Always assign this network device the same IPv4 address. If you tick that the address never changes. Also in modern Fritz!Boxes port forwarding is associated with a particular host, so I think it also works without the static assignment enabled?

Anyway, I have logged on to my headless GPU machines remotely through port forwarding for years and never had an issue.

In the US when a device is "on the fritz" it is failing intermittently, and the classical solution is to smack it firmly until it works. I suppose a Fritzbox might be perpetually on the fritz.

Same, I have used Fritz!Boxes for years, they are reliable, get updates and are quite configurable. The labs version even has Wireguard support now (they had IPsec before).

Do you not have any hobbies? I find this to maybe not be practical, but that’s not the point of it.

Everything has a purpose, unlike many "home labs" where people are just tinkering. There's nothing in here that would require fussy maintenance. It seems pretty reasonable to me given the functionality.

If they think this network is convoluted they should see mine!

In my experience, the main issue with setups like that is IoT/convenience devices being subtly broken because of all the firewalling. Then you suddenly find yourself trying to figure out why you can't just airprint from your ipad or why your guest's iphone sees a HomePod, tries to activate airplay, but it just silently fails. Really fun to debug, especially when you need that document printed right now or when you have a party going.

But what's the alternative? Unsafe home network where one rogue device can act as a tunnel for bad actors(bots more often tbh)?

If you buy devices from trustworthy brands and replace them when they stop getting security updates, it should be fine, right? After all, aren't 99% of home networks 'unsafe' according to your definition?

>After all, aren't 99% of home networks 'unsafe' according to your definition?

Prevailance of home ip addresses in DDoS attacks and in proxy pools does suggest so ¯\_(ツ)_/¯

It doesn't follow. There are a lot of homes, so even if 1% of all home networks had "rogue" devices in them they'd dominate DDoS attacks. Besides, it's not HomePods or Withings smart scales or Hue bridges doing that as far as I'm aware, it's mostly cheap, unsupported, noname crap, so you can reduce your risks substantially by not buying questionable products.

There are plenty of CVEs in brand name things across IoT spectrum.

Vetting devices you introduce to network is of course solid advice, but a little bit of paranoia never hurts in tech.

How many of those get exploited on firewalled networks before they're remotely patched though?

My whole point above that it does actively hurt, with devices randomly misbehaving at exactly wrong times. It's not enough to set up everything once because devices get updated and change ports, domains, and protocols. It also makes everything more brittle, requiring multiple inter-VLAN proxies to be running at all times for seemingly unrelated devices to work. That SD card in your raspi died? You decided to update Docker on it and run into problems? No Sonos for anyone in the house until it's fixed.

There's a real cost to that paranoia, it's just another case of security/convenience tradeoff.

Let's agree to disagree, I think in the end it comes down to priorities and pain threshold for having to tinker with stuff.

The alternative is roughly what google called BeyondCorp — not trusting your network and doing explicit auth everywhere it matters, maybe with a sprinkle of Tailscale to simplify auth and encryption.

If you're worried about your network being saturated for DDoS by a random IoT device, I suspect you'll notice it even without explicit monitoring.

Besides, risks need to be weighed by their probabilities. It's a small chance of name-brand IoT devices "going rogue" vs the certainty of random things not working when they should, and I don't think this tradeoff leans towards VLANs for most people.

My home cactus garden has an unnecessary number of cacti in it, as compared to the average home. I also expend unnecessary calories when hiking to places I don't need to go.

(edit: admittedly the five or six times I've setup a home network more complicated than just connecting to a router I've ended up regretting it after a few months)

Sometimes I even just walk in a big circle and end up where I started! What a waste of time!

Building my home network though is teaching me IPv6.

Sure, you can use the ISP modem and a laptop on wifi.

But that sucks ass.

Wouldn't you rather have real monitors/screens, a solid wired connection to a network and a real keyboard and mouse? Yea it takes space and time but its way better.

> Wouldn't you rather have real monitors/screens, a solid wired connection to a network and a real keyboard and mouse? Yea it takes space and time but its way better.

I do for most things, but better is personal.

Saying that OP's setup is overly convoluted or better is entirely missing the point -- it's what they want to do for enjoyment. Personal taste doesn't need to be justified.

Direct hit to the heart *cries in BGP and big enterprise switches*

Reply on news.ycombinator.com