Why I quit this battle
badmodems.com
badmodems.com
I ended up making a big stink elsewhere and they got the repo down. Funny enough, their heads of security told me they'd use my disclosure to push the execs into building a big bounty program. Long story short, their CISO told me on the phone that what I found wasn't a "bug", and that if they did a bug bounty program, they'd go bankrupt.
Ha! Classic Telco. I've seen some in my country and they are an impressive mess of legacy (and many times redundant because of all the M&Âs) applications and undocumented integrations made by an army of low paid outsourced integrators.
Also, low effort mode overall, like your CISO friend there, who probably just wants to survive for sufficient time too jump ship.
His bug bounty speech doesn't hold, as they can start with a very low bounty and increase over time to get the interest of higher skilled people and reach more complex bugs, having total control over spending.
Also, Black hat experts probably have those already mapped and are selling them to the highest bidder, and with privacy regulations getting stricter that "bug bill" will come to them sooner or later.
I believe it. When I worked for them a few years ago, their internal security was pretty bad, and they had tons of random teams with no security guidance, governance, etc. I think the only reason they could operate at all is nobody is trying to hack them. It might be a little bit better now, but knowing the scale and state of things, there's no way they've magically knitted everyone up into properly managed AWS Organizations, to say nothing of actually supporting individual teams' security needs.
The "good news" about your discovery is that it probably was limited to just one tiny system, because everyone maintained completely independent systems and didn't have access to anything else - not because they weren't allowed, but because you didn't even know what other systems there were, much less know how to request access, and virtually nothing internally used SSO. The only way to learn about what other systems there were was to walk around the floors of the Comcast Building and ask random people what they do.
I have no sympathy for these companies. Fix your shit, secure your users. Don't? Well I guess I'm posting these non-bugs all over the net for criminals to enjoy.
I appreciate the honesty! I do think that any company that offers a bug bounty program already has their security in order, to the point where they can no longer find any obvious issues themselves anymore. Not having a bug bounty program implies they don't really trust themselves yet, or haven't reached that level of maturity yet. That probably covers most companies though. I know the codebase I inherited at my current employer wouldn't pass even the most superficial security check. Its only line of defense is that it's only on private networks. Until it isn't. I wonder if I should do a google to look for any public instances... just did, I'm only finding a lot of search engine spam thankfully.
This is why the bad guys win so often. Even companies that do pay bug bounties often pay very little compared to what exploits could be sold for.
I was given a modem by my ISP that had a Puma 6 chipset and it was a nightmare - it would completely cut out multiple times per day, latency was all over the place, and it was a truly terrible experience.
Doing a bit of digging I was able to short-circuit a lot of troubleshooting and replace the modem, which immediately solved the problem. It truly was negligent how bad those chipsets work - even for basic tasks it was nearly unusable.
Imagine hundreds, or thousands of houses out there all with 25+ year old coax jacks in walls, 4-way splitters jammed into ceilings, leaky outdoor connectors that have been chewed on by squirrels, etc.
It's a miracle that DOCSIS3 works at all.
I'm vaguely familiar with the issue that the axe-grindy guy in the original webpage referenced here is complaining about, specific to some series of chipsets used by a popular cheap cablemodem for a few years of manufacturing run... But what he's found is actually a tip of an iceberg of terrible about why maintaining last-mile copper last wireline infrastructure (POTS wiring/DSL or coax for cablemodems) is a losing bet in the long run.
All copper in my city is replaced by fiber, and only copper is from the FTTx box in front of my apartment to my flat. Eeerything (landline, internet, IPTV, etc.) runs from a single, dark fiber.
With cabling already in place and VDSL can reliably provide 5 units of bandwidth (where 4 units is used for download and 1 unit is used for upload, e.g. 32/8 mbps), with a theoretical ceiling of 350mbps, I'm pretty happy with what I have.
While I'd love to have fiber at home, I definitely don't want to rerun cabling.
Also, the term looks like it has both meanings.
Having a "dark fiber infrastructure" means having a web of unconnected fiber cables point to point, but since you connect your own devices to it, you don't have to share it with others.
As a result, its presence and its state is only known to you (i.e. it's in the dark).
I understand that my usage is not completely true, but in that context, it's not a flat-out blatant mistake either.
Even the name means: "it's dark, no light, as in no data, because that's what light is in this case"
It has two meanings. I have heard it most commonly used in reference to private fibre networks.
I can't wait until my HFC NBN service here goes the way of the dodo.
While I have 1000mbps down, the 50mbps upload is just miserable. And the cost is painful, $130 AUD a month.
For the price I pay I should be able to have a shared 10000/1000 (in reality it will probably be 2500/1000).
All of this for the swag, I am ,ot likely to use the bandwidth anytime soon (despite serving a few things from home)
Now at least I can play my video games while my partner watches 4K shows on our TV.
Interestingly, moving from 100mbps to 1000mbps has given me a 5ms lower ping: 31ms to 24-25ms with a decrease in jitter of 5ms too, in Valorant -- this is to the exact same server from the physical same PC. I'm guessing even on the same damned cable, the 1000mbps customers get some kind of routing priority or something.
This makes it the best option by far for communication in hour-long blackouts when, for instance, cell phones and cellphone towers would have ran down their batteries.
The great thing about this was they could circumvent telecommunications law that requires them to rent out circuits at a competitive rate to small carriers, basically putting the nail in the coffin of small, independent DSL ISPs, since the copper loops are gone. And you, the customer, are now stuck with 6Mbps DSL from a single provider for $80 a month.
Almost everywhere in the US you can get at least 25mbps for less than $80 a month. And within almost all cities and their suburbs, you can easily get 100mbps+ for less than that.
Unbundled network elements was a stupid utopian idea to begin with. No one who knows a lick about broadband policy actually wants to go back to anything like that. The current model is essentially that you're responsible for building your own infrastructure if you want to be an ISP. That has actually spurred an incredible amount of investment in infrastructure across the country, since you know that if you spend a ton of money laying a thousand miles of fiber, you'll be able to reap the rewards of it, not have to rent it out at cost to your competitors.
American internet service is the best or one of the best in the world for a large, populous country. Certainly, the major European powers don't come close (remember when Europe had to beg Netflix and others to reduce video quality to 720p early in the pandemic? US internet service held up perfectly without any throttling.). Only Japan and large parts of China are comparable (again, not counting small or lightly populated countries).
There's a lot more to robust internet service than what you get on speedtest.net or in optimal conditions. Further, the internet speed statistics show that European internet service is on average slower than American service. It's great that you good service, but the point I'm making is that American service is on average faster and more robust, and we got there by abandoning the unbundled network elements nonsense.
But even if speeds were equivalent or better in the US, prices in the US are still multiples of European prices for the same service.
Also what's available and what I'm purchasing is different. I'd pay about 35$ for uncapped 100/100 and that's enough for me. But everywhere where you can get 100 you can also get 1000 for maybe 90$ a month. I don't want 1000 because it's useless for my use case, and I don't have anything to prove so instead of wasting money I'm staying on 100, I think most people are because it's just enough. There's also 250 and 500 inbetween which some might get too.
This is Sweden, which is supposedly 50 Mbit behind the US (x for doubt)
It's a beautiful ideal to reach for, but it's expensive, because it implies actually investing in infrastructure instead of exec bonuses.
We're also apparently building out train infrastructure quite extensively in Europe, and when you lay down train tracks you lay every other piece of infra down with it.
I'm all for private actors and all that, but not when the actors are allowed to pay politicians to not regulate them.
- 4 different fiber ISPs (in practice only 2-3, because 2-3 of them are sharing the same fiber line)
- 1 ADSL on the copper line
- up to 4 5G connections
- who knows how many satellite connections
That would be total overkill for 99+% of the people of course, but it's really nice to have 0 downtime when switching ISPs because the previous one still works when you start using the new one !
And regardless, while that might be a useful property in certain circumstances, the copper-based NBN we've ended up with is so woeful, that they're basically declaring technical bankruptcy and moving to FTTP after all that in the end anyway. Just a decade late and with billions down the drain for sub-par expensive flaky internet.
I can run 1 Gbps symmetric for ~48h without power by running my optical network terminal (ONT) and a teeny travel router on battery packs. (Used this set up through several long power outages.)
all the above describes my apartment connection in Astoria. and the neighbors were DIY tapped into it too. At any given time I lived there there was an appointment scheduled or being scheduled to have a field worker try to fix the downtime issues. they never did and i moved out
The OLT can often pick out which ONT is doing it.
> Some chip makers have hidden latency and jitter issues from common tests that are in use by consumers and even ISPs. Ping CANNOT be used reliably to test for latency or jitter.
https://web.archive.org/web/20210806220209/https://badmodems...
> The Intel Puma 6 and Puma 7 chipsets have a performance issue in a part of the packet processing that is specific to TCP/IP, which went undetected for awhile because "ping" packets are not TCP/IP. [...]
(And/or because just running "ping" a few times might not catch the issue if it doesn't happen at the same time as the lag spike.)
No idea if it happened in this case because this is the first I've read about it, but it has always struck me that the websites of campaigners get progressively worse as the battle gets more and more difficult, and fellow campaigners drop out, leaving the diehards with both the campaign and maintaining the overly-ambitious campaign website that someone earnestly set up. There always comes a point where someone just hacks the HTML by hand.
Either way, I wish the guy peace of mind and hope someone can help him with his medical bills.
Sometimes you're just done. Makes sense to be clear to everyone involved and put it behind you.
> Then,, KevinDS on DSLReports was just a incredible ass to me. It was epic. I believe he also lied to me in PM when i tried to get him to help. I was already unhappy with the future that I was looking at. He then just said nasty things. I was done.
Seems to be partially self-inflicted ?
xymox1 wrote :
> You know morons like me should not use Wireshark....
kevinds responded :
> I was thinking more along the lines of,
> Morons like you shouldn't be telling the grown-ups how to secure their networks...
> Sorry, not sorry.. ;)
Now the "grown ups" part is clearly not nice (but then what do you expect when you're starting to lay blame even though you don't know how Internet networking works?), but the "moron" part is clearly mirroring his initial self-deprecating remark, rather than a direct insult.
Also, as cramer says :
> (If anything, he doesn't understand the magnitude of the windmill he's charging.)
----
Wait until he learns how consumer routers typically have the IPv6 firewall off by default (if any), and there's no (need for) NAT (which has never been supposed to be used for security anyway, and also properly set up IPv6 suffixes are too big to be scanned), so "internal/home" devices are "directly" on the Internet !
Mirroring someone's self deprecating remark, adding an insult, and adding "sorry not sorry" is being an ass. Or at least, it takes a lot of familiarity before it is not being an ass.
Society progresses, one battle at the time...
Thanks for all the fish!
On the other hand, it seems the issue that was the straw that broke the camel's back is saying that ICMP equals access, and access could mean just ICMP to some (I can access this thing via ICMP), or access could mean full takeover to others (now I know it's there, and therefore it's exploitable to anyone with time & energy).
It's really stupid to get upset about self-proclaimed security experts arguing about stuff where they haven't even clarified what they're arguing about.
Honestly, though, it's a non-issue. Some ISPs use RFC 1918 addresses, and some of those addresses are reachable (via ICMP) from consumer endpoints. So what? That's no less and no more secure that any other set of intermediate hops in ISPs' networks.
https://news.ycombinator.com/item?id=15781474
> Some chip makers have hidden latency and jitter issues from common tests (badmodems.com) 220 points by based2 on Nov 26, 2017 | hide | past | favorite | 42 comments
So I am the Badmodems.com guy. I started getting emails to the badmodems email and I came looking for this thread.
I did fight the good fight. I took out a whole division of Intel. They sold off the connected home div and the Puma. Then the Puma has now pretty much died out.
That last battle tho, with the cable co admin network. OMG. That was some crazy shit. My health is better now and putting all that crap behind me was the right thing to do for my sanity..
My fav part of the whole Intel battle was when i did a 6 hour deposistion in the class action lawsuit. Arris lawyers and Intel battering me for 6 hours. All under oath and videoed. That was SO AWESOME. It was a intense battle. I won all of it. They never pinned me down once. I can also now discuss it and Intel did internal testing and even hired a outside testing lab. They did this really early. The discovery process got emails that confirmed horrendous performance and in email they decided to keep selling it with defects they knew they could not fix.
The cable co admin network thing,, oh gawd what a mess. There was a HUGE comcast network faceplant right when I was working with a guy in europe and with Comcast. Comcast as a whole pretty much fell over in most cities. I never found out what happened. I think Comcast was trying to patch holes and borked their own network.
I am glad to hear people got something from all my efforts. I did save everyone on cableCo systems worldwide from the Puma/Intel. They literally had plans to dominate the world and take over the home while destroying the only competition, Broadcom.I did, pretty much single handed, prevent really nasty devices from polluting the world and now have sent that whole chip into a grave.
The code that makes these devices fail uses up lots of UDP ports which suggests that the issue may be with the NAT implementation. I have done some searching online but haven't got a definitive answer on what exactly the issue is (other than it causes latency and hitter under certain conditions).
I'm slightly sceptical these modems (really modem routers) are as bad as this site previously suggested. It seems like the only use case that breaks them is some specific games where there is a huge amount of latency critical UDP traffic.
http://web.archive.org/web/20211002154145/https://www.badmod...
---
I feel bad for him. Fighting for good security or even a better product shouldn't cost people years of their lives but as long as it doesn't come at the expense of the product's bottom line, there is little reason to compel change as it doesn't impact the bottom line which is what decides whether something is made for most big corporate outputs.
The same argument could be said about "Tech Debt" or non-UL labeled products and a myriad of other causes.
Zero fucks will be paid to anything not socially cool even if it is the responsible thing to do.
That's why we live in the world we live in now.
It doesn't take a rocket scientist to figure out why the world is crap. But for the most part, it takes that "title" just to have a microphone loud enough to move the needle.
This little part of life is what I'm currently struggling with in my post-Covid decision making.
The site seems to be focusing on latency/jitter issues: https://web.archive.org/web/20210506114456fw_/https://badmod...
I'm not entirely sure what to make of the information presented. The page above includes an embedded video demonstrating a difference between ICMP and TCP load-testing: https://www.youtube.com/watch?v=15cJ400yR_E
I'm not knocking that there is interesting data, I'm rather trying to consider potential confounding factors that could call the conclusions being drawn into question.
In particular, I have two questions:
- This doesn't talk about "bufferbloat" (https://en.wikipedia.org/wiki/Bufferbloat) or internal buffering happening inside the modem, only latency/jitter. I'm honestly completely naive about the relationship between the two concepts, but I thought they had reasonable overlap, and could even potentially go some way to explaining the dynamics of what's going on. Why not?
- The video seems to just be TCP- and ICMP-pinging 8.8.8.8. Surely that address is sufficiently hammered that it probably implements fairly aggressive rate control algorithms...?
"Then,, KevinDS on DSLReports was just a incredible ass to me. It was epic. I believe he also lied to me in PM when i tried to get him to help. I was already unhappy with the future that I was looking at. He then just said nasty things. I was done. It would be a thankless battle I would wage on my own and I was unwilling because of people like him. So KevinDS take a bow. CableLabs loves you, Intel loves you, Arris loves you.. I was a huge thorn in thier side. The Badmodems.com site gets 700 uniqui IPs a day. 380,000 unique since the start. 16 million hits. So KevinDS, you pushed me enough at the right time to end all this. I will NOT spend the next 5 years in a new battle that is thankless and filled with people like you."
The content above doesn't belong on the front page, simply put. It's very negative, and I'm sorry that I just continued with the negativity.
You might think "that only moves the attack surface" , you are both right and wrong. Technically it is moved but you eliminate LAN based, wireless (think wpa) and untrusted network devices from the equation and reduce them to one attack surface. Not only that, the threat actors most relevant to most people's threat models are not able to operate or operate with reduced capability to the most part when the attack surface is not a home network/device managed by individuals (and crappy vendors,isps,etc...).
It can pay for itself, don't worry about network device security, just get the cheapest yet fastest stack amd VPN through it.
Whereas securing your own network keeps them intact within your infrastructure.
Cloud is not always the answer. Learning to network and compute, is.
This is why we should learn from what Ed Snowden presented as just about all of the data gathered by NSA and their contemporaries utilized the concept that encryption and security should only exist at the edge and they were capturing network traffic inter-network and intra-DC.
There were reasons why companies like Google took his whistleblowing seriously and changed their infrastructure accordingly because none of them feel safe leaving their "secret sausage recipe" exposed while still technically out of band from the customer paths.
I truly hope they can feel proud, without any guilt.
"Badmodems.com is worth something, so, with medical expenses in front of me, its time to cash it in"
I think his efforts to publicize the issues with bad modems was worth something to modem users.
But there's nothing to cash in.
He's selling the domain name
No idea what his most recent issue is but hey, thanks for your work bud, it was useful to many. Time to take a break!
https://web.archive.org/web/20211002154145/https://www.badmo...
The badmodems guy seems to be upset that he is outsmarted, honestly. Thanks to him for the good he has done, but if he can't handle being in a room with others smarter than him, this departure is for the better.
He seems to know what he's talking about, but is also invested enough to become a potential sell out.
But the entire thread was hard to follow on that site. There wasn’t much context.
For example:
I was well aware that you needed to get a MAC address whitelisted and then from there a config, including your speed tier would be pushed. I somewhat assumed DHCP reservations were used for that, on a management network, with TFTP for the config file. It makes sense.
This was also solidified by that fact that in most cases during an internet outage, my routers IP would revert back to an RFC1918 address.
Ive never used "landline" services from these providers so SIP wouldn't really be in scope but its not surprising that would be on a separate network and there may be some ability there to "sim-ring" calls. Thats standard in a lot of SIP implementations and probably a feature even a home user would want (ie: ring my cell phone and allow it to connect if phones on the modem dont work).
Things like redirecting traffic etc would get noticed with a ton of stuff being SSL/TLS encrypted. But i would agree some/a lot of this should be using TLS as well.
That said I have never mucked with any of it because.
1. I consider the modem, which i own, to be untrusted and while its inside my DMARC, its outside my security perimeter. Its not directly managed or controlled by me. Anything on it or passing through it should be considered hostile and subject to inspection or filtering.
1a. This is 100% of the reason why i run my own firewall, separately, from a device managed by the ISP and why i avoid AIO style modem/router/firewall devices.
2. I am unsure what mechanisms an ISP may employ to ensure certain things (like upstream/downstream configs) are not tampered with. This could be as simple as a hash check monthly or when billing rolls over to ensure my version is the same as theirs on their servers. Changing that could get be out of bounds with the TOS and canceled, which i have few other options.
So its incumbent of me to basically mind my business. I can also see me "unleashing" my speed tier could impact others on my node, which may cause calls from other customers and an investigation shows that theres a sudden over subscription outside of their norms/standards. Again could be considered malicious and cancel my service/blacklist me or the address.
WITH that said, i do understand the concern with respect to AIO style devices. But again i would consider anything on the ISP network to be "red" and no different from the relative hostility of the greater internet. But i dont see the concern with DHCP traffic and arp traffic, that seems normal, even on a ISP net, its how devices get online and authenticate and find the next hops on the network.
ISP's should do better about segmenting that though, and should probably not provide an AIO solution in general or if they do have one with actual phsyical segementation (ie a box with 2 boards independent of each other connected the same way a modem and router would separately) but I understand why they may want to have simpler setups as well from a customer support standpoint considering the average technical prowess of their userbase.