Upgrading my Networking Gear (2024.1)
This is a bit of a smaller update while I wait for some equipment to come in, but since the last bit, I’ve made some hardware upgrades which showed up on my network cart rebuild so I figured I would talk about. Even though there’s never really such a thing as ‘done’, I’m slowly getting closer and closer to that state (from a hardware perspective).
Networking Series
This post is part of a series, check out the other posts!
Quick Links
Starting point
In my previous update, the focus was on establishing a secondary, low-power network that I could run during an extended power outage (say, an 8 hour work day). With a sub-30W footprint, this isn’t a hard target to hit at all. I actually have some a process in place to be able to calmly transition the network rack from utility-power to use battery-based ‘generator’ to support longer outages (because, priorities right?).
Powering my network stack (and more) during a power outage
Over the last year, I’ve made a few hardware upgrades and there are a few specific problems that I’d like to tackle. In no particular order:
I ended upgrading to a 10G router. I don’t need it yet (ISP capped at 1G symmetric), but when the times comes for faster-than-gigabit, I’d like to have everything already in place to go. There is a benefit that I can realize now — the vastly improved VPN performance
My outdoor AP sort of died so I used the opportunity to swap it out for a WiFi6 model
During the summer, there was a one day sale on a high-density AP so I bought that to act as my new ‘main’ AP and transitioned my current one to be dedicated to my low power network
The batteries in one of my UPS units hit their end of life which got me thinking (more on that later)
New router - TP-Link ER8411
I’ve wanted to upgrade my router for a while now. Originally it was to get ahead of a potential service upgrade from our ISP but more recently, I’ve wanted make better use of my current connection. In particular, I wanted to make better use of my available upload speed through more modern VPNs (OpenVPN, SSL, Wireguard, etc). We can see a monumental performance uplift when looking at the 10G router.
Yes* indicates that the featureset is [supposedly] supported but I wasn’t able to identify a reliable source for the performance metric.
In a vacuum, the ER707-M2 is actually an interesting choice: it gets me 2.5G WAN/LAN capability and a respectable performance numbers all around (in particular, OpenVPN performance) without breaking the bank too badly. However, since I’m coming from the ER7206, the performance jump is less impressive and so, the 10G ER8411, even with the massive price tag, starts to make a bit more sense as a meaningful upgrade.
This upgrade has one additional benefit: the ER7206 is not natively rackable: I do have a 3d-printed rack-cage for the ER7206. One catch is that the specific 3d-printed mount I ordered was bigger than 1U, which threw my spacing out of whack (and this has been low-key bothering me since I got it racked up). It’s an OCD thing.
During the upgrade, I took the opportunity to try and eliminate as many SFP+ to RJ45 adapters I had in use as I could, swapping them out for DAC cables. Why? Because the DAC cables don’t need to perform any signal conversion. This has the benefit of having the least amount of latency but also the least amount of power consumption and heat. Before I started racking up networking gear, I didn’t think that it would make a meaningful difference, but seeing the temperature difference with DAC cables is eye-opening:
15C difference (steady state) difference between connecting via DAC versus running through a transceiver.
On the ER8411, the 10G WAN port is SFP+, so if my ISP ever provides faster-than-gigabit, I will likely need to run a transceiver; but for now, I opted to use one of the 1G WAN ports and connect to the modem via CAT6. While a single module isn’t going to make a huge dent on the room temperature, if I can avoid extra heat, I might as well.
To make the [future] transition easier, I included the SFP+ when making router-rules; so in the future, I should just have to worry about switching the physical connection and not having to update any of my ACLs.
Why not something like OPNSense?
This is something I’d like to revisit down the road but at this time, truthfully, it’s not in my current skillset and comfort zone. From a hardware management perspective, I wanted all of this to be accessible from a single portal/app. Perhaps, down the road, when I outgrow this, I may look at something for serious.
New access points
My outdoor AP started to develop some weird behavior where it would broadcast SSIDs and allow devices to connect but would absolutely refuse to provide a connection to the internet. Indirectly, this failure ended up being a selling point for having everything on physical switches — Nicole was able to power down the AP without fuss (and thus, connect to any of the other APs that were working correctly). I did not want to deal with calling into support to troubleshoot the issue, so I just opted to replace this with my first WiFi6 AP, the EAP650 Outdoor.
Over the summer, there was a sale on the EAP 620HD and even though I was realing pining to get the EAP 690e HD, I just couldn’t say no to the price. The model I received was a V2 (rather than the latest, V3) so I missed out on two significant improvements:
Approximately 60% smaller footprint (290 sq-in vs 125 sq-in)
The V3 can take passive-POE and includes an injector; the V2 can only take active-POE and doesn’t include one
Thankfully for my scenario, neither of these downsides really affects me: I have the AP mounted in the furnace room, out of sight and for POE, I picked up a cheap TRENDnet POE Injector. With this new ‘main’ AP, I was able to transition the previous AP to my low-power Endurance network. I did this by ‘forgetting’ the previous AP from the control panel and then configuring it using standalone mode. In standalone mode, I configured the same SSID/password combination so things should mostly work as expected when cutting over.
This leaves me with three remaining WiFi-5 APs (all EAP-235Wall units) and barring a fantastical sale of some sort, I don’t see myself upgrading any time soon.
Dual DNS
Redundancy
A few updates ago, I set up an ad-blocking DNS server on my network and it’s been pretty great but I’m always super wary about the robustness of the Raspberry Pi (or maybe it’s Linux, lol). Everything works great — until it inexplicably doesn’t and it’s not a quick fix because you can’t even remote into the machine because it’s got a kernel panic. In fairness (and definitely hindsight), I suspect that the kernel panic was caused by ‘not enough power being delivered’:
There isn’t really a nice and easy way to verify that now and
Frankly, I don’t really want to deal with it
Outages are never at a convenient time!
To mitigate this, I bought a second Raspberry Pi — I went with an identical RPi4 unit to what already had. The plan was to just clone my existing system image, tweak a few things to change ‘Pi1’ to ‘Pi2’ and that would be the end of it. For once, I am super happy to say the process went exactly that smoothly.
When I rebuilt the network rack, I made to sure add USB ports (connected to each Pi) to the front side of the rack. This way I can do fairly easy system backups by just connecting a card reader or SATA-USB dock - I keep two full backups on standby. In the event of a device failure, it’s a matter of just unplugging the existing boot drive and plugging in the new one.
Syncing and Caching
Once both Pi units are up and running, it was simply a matter of having the primary and secondary DNS servers for my whole network point at them. You could stop here and be done: it’s totally possible to manage both Pi-Hole instances individually but it would be much nicer to be able to have [config] changes automatically sync between them. This is where Gravity Sync comes into play.
Gravity Sync was super easy to setup and verify; it sync’s blocking rules, exceptions, local DNS settings - everything I would want it to do. I’m still responsible for doing updates on the Pi units but one nice thing about having the extra Pi, is I can stagger my updates: I can update Pi1 more often compared to Pi2 with the safety net that if an update goes wrong, I have the Pi2 to carry the network while I restore a backup.
One thing I did learn was that ‘primary and secondary’ DNS servers are automatically load-balanced (to a degree) by the router. I had previously thought that the secondary was really a hot-spare and only really came into play when something went wrong with the primary. The load-balacing definitely favors the primary, but the secondary certainly doesn’t sit idly by.
Query & client distribution snapshot between my two Pi-Hole DNS servers
While I was in the headpsace to be working with this, I opted to setup a recursive DNS on one of my Pi units - I figured that if something went wrong with the recursive DNS lookup, at least the other Pi can perform ‘normal’ lookups. The motivation here was less "'privacy concerns’ but more ‘self-reliance’ and the possibility that the local-recursive DNS server would be faster (with the caveat that the first unique inquiry would be slower).
Comparison between query-caching distribution of recursive (left) and non-recursive (right) DNS
I don’t know if there any conclusions I can draw from any of this — it’s just neat to see! Obviously, Pi-Hole does have some localized caching but there is a non-insignificant number of queries that it has to ‘reach out to Cloudflare’ to resolve.
Looking ahead
I’m in a holding pattern, waiting for some hardware to come in which should sort out the last bit of my network stack. Aside from that, I need to spend a bunch of time handling the configuration aspect of everything:
IP address management, documentation and diagramming
review of access rules, permissions, etc.
review of backup strategy
On a related tangent, I’ve started exploring the rabbit hole that is self-hosting services…
Network Update (2022.4)
In the previous installment, I spend time to spin up an entire network stack as a (low-power!) cold-spare. This way, if the power goes out, we could conceivably maintain mostly-normal operations.
Product links may be affiliate links: MinMaxGeek may earn a commission on any purchases made via said links without any additional cost to you.
