Hacker News Reader

Comment by ChuckMcM | Hacker News Reader

I've seen some of the papers where people look at big chunks of unused address space and watch the probes etc. It is really quite amazing. Once I screwed myself royally by accidentally turning RIP on for the upstream side of my router (connected to the cable modem) and it advertised 192.168/16 which Comcast accepted and started routing random stuff from the local exchange to my router. It was pretty funny talking to their NOC staff who was mad at me for advertising the route but I pointed out it was pretty stupid of them to accept routes like that from their edge nodes. What if I had sent them a BGP route for China, then what would they have done? (Don't try that at home kids unless you don't want to use your network for a while)

View on HN

As an ex-Comcast employee. Seeing stuff like that happen really doesn't surprise me

When Comcast first rolled out that data cap nation wide, I started prodding at it one night out of morbid curiosity

Turned out that it would silently slurp all HTTP traffic! Once you hit some arbitrary measurement (EG: 50%) it'll immediately start hijacking all HTTP websites you visit and inject a ton of Javascript to put a message over the web page forcing you to acknowledge your cap

Nmapping the server they used caused the messages to immediately disappear. As well as the server to seemingly vanish. Turned out the firewall was just blanket banning the entire IP range when it saw a portscan!

The upside being that Comcast would stop MITM'ing HTTP traffic for about 72 hours

And people wonder why https everywhere is such a necessity now. It should not be necessary to treat your last mile ISP as a hostile entity , but sadly, it often is.

I canceled my decade old COX account last time that happened. Even asked nicely not to "help" by editing traffic, the runaround was fun. At the end they offered to take my ~$90/mo to ~$70; re-confirming they had no idea what I was unhappy about.

A $12/hour call center customer retention worker in rural TN has no idea what you're complaining about, their job is simply to meet some retention metrics on a weekly basis.

Even if you can actually reach the people who run the ASN of your ISPs, if it something big like Cox, charter, Shaw, etc, they'll be politically unable to confirm or deny anything, and won't want to talk to you. You might get a straight answer if you are in a similarly senior position at an equivalent sized ISP that has mutual settlement free peering, such as between RCN and Charter.

Such are the joys of modern "customer support" -- human beings don't scale, because they need to sleep and can only talk to one other human at a time. So you hire the cheapest ones you can find, and instruct them to be minimally helpful. Even better, make them all "managers," so "can I please speak to your manager" will just take you to another minimum-wage employee.

If you want actual customer support these days, your best bet is to create a PR problem for the company via social media, because PR flacks are paid enough to matter.

In cases without competition. I live in an area with three gigabit-capable ISPs and the difference in support quality is unbelievable, even for the Comcast and Verizon customers – just calling from a competitive neighborhood gets your call processed differently.

Which city?

Sounds like Seattle

Washington DC - my neighborhood has Comcast, RCN, and Verizon. Our neighbors report better service on all three — and when you go a couple blocks south where the FIOS rollout stopped, regression to normal sets in for Comcast.

We also have municipal fiber but they’ve chosen not to make that available for residential service which is really disappointing but … politics.

So, channel bonded docsis3.0 and 3.1 on Comcast and rcn, and gpon fiber from Verizon? Or are the cable operators also doing singlemode to the house now?

Definitely for RCN. Comcast ads have claimed fiber but I don’t know whether that’s available or just planned since I won’t do business with Comcast.

A smarter, and perhaps even more profitable way to approach this problem is "OK, we want almost everything that happens to customers to be a self-serve situation" and then make your (fewer) call centre employees highly trained troubleshooters who can figure out why _this_ customer wasn't able to self-serve and get that sorted for them. The result is a better customer experience (usually everything just works, when you need to talk to a human the human is an _expert_ who helps fix your problem, not a drone working from a fixed script) and it can be much cheaper if done right.

Definitely if you think of your product/ service as "premium" this is the correct model to have.

Lots of people simply do not like self serve. I get that companies want to cut costs, which is why we got endless telephone menu trees, then support web sites with crappy search, then automated telephone agents, then chat bots, and on and on. And the public has devised all these various tactics to skip that crap. I just want to talk to a damned human being rather than spend 10x the time navigating your poorly thought out Customer Avoidance Systems! Sometimes I need a drone and sometimes I need an expert. I have yet to see an automated system that could successfully determine which tier support I needed.

Often I prefer "self-serve," but sometimes I like to talk to a human who knows about the system that has created my problem, and can quickly fix it. That's why there are websites dedicated to showing people how to skip through phone menus and reach actual humans, even if those humans are soundboard operators paid almost nothing to make you go away.

link? I refuse to talk to computers.

The fact that these pages exist tells you all you need to know about the fundamental user-hostility of modern tech companies:

https://gethuman.com

https://www.wikihow.com/Talk-to-a-Human-when-Calling-a-Busin...

Impressively (given the rest of the company), Comcast DNS is on twitter [1] and is fairly responsive/tech savvy and unrestrained. It feels like someone in the department got one over on management - "look, if you get in our way things break real fast. Trust we know what we're doing and you won't get calls about state.gov being blackholed".

1. https://twitter.com/ComcastDNS

Your last mile ISP is almost certainly a hostile entity, but HTTPS alone isn't going to save you [0].

[0]: www.cs.umd.edu/class/fall2017/cmsc818O/papers/tangled-mass.pdf

I suppose the only alternatives are to use a VPN (and hope your VPN provider isn't a hostile entity), or to use TOR (not appropriate for all kinds of traffic). This is the world we've made for ourselves...

That paper is from 2014 and some of the circumstances described have since changed.

They've also done some things that I assume fell out of operating ICSI's Notary but don't make any real sense for this paper.

For example: For a real user what we care about is this cert the end user was presented for a site: Would that be trusted in (Internet Explorer on XP, Safari on iOS, a Python script on a Debian machine, etcetera) and would it be trusted in this smartphone.

And what they've looked at is, were the same Trust Stores baked into an Android phone as the above systems? But that's subtly different in a way that fogs the issue here. Example:

Suppose phone X trusts ISRG Root X1, XP trusts DST Root CA X 3, and a Debian system trusts Lets Encrypt Authority X3. Those are, to the naked eye, and this study, three completely different things. But in _practice_ for an end user it'd turn out any of the three work for trusting a vast number of certificates used on the web. Trusting one or another _does_ matter, but this paper isn't about why that is, and doesn't really explain what's going on here, it treats that sort of scenario as anomalous and potentially alarming without explaining.

The paper did remind me that ICSI's Notary won't work with TLS 1.3, which I have sort of known but not ever mentally addressed. The ICSI Notary works by peeking inside TLS sessions. In versions up to TLS 1.2, the server's Certificate is delivered unencrypted, just before both peers encryption switches on and their communications are unintelligible. This is used by the Notary and by lots of crappy middleboxes, but in TLS 1.3 the encryption has switched on earlier, before the certificate is sent, so the Notary can't see certificates any more.

At my work, we treat our own internal network as a hostile entity. Defense in depth.

Same here. Devs can't even mount thumbdrives. Which is fine for us, we don't need them, but it prevents some funny business should one of our laptops get stolen.

Ditto. No flash drives except for special ones distributed by the IT department, and even then it takes a few months of reviews to get one.

But we go one further — no development on the LAN. Develop on another server at a different hosting company, and only deploy on the production host when thoroughly vetted.

For some reason, I get this mental image of everybody in a city street dressed in hazmat gear.

> no development on the LAN

Does this just mean staging/shared servers are not deployed on machines inside the corporate network, or that developers can't have their dev environment locally and instead remote in to some other machine to do their work?

I've heard of the latter in a few companies and it's always the kind of thing that makes me nope out of every applying to them.

Devs do most of their work on localhost. Changes are pushed to an independent server on an independent network in another city. Once those changes are vetted, then the work is mirrored to production.

Each dev has his own cellular connection for internet on his development box. Again, no LAN access. Corporate e-mail and cross-department file servers are on another box on each dev's desk.

The downside is massive over-usage charges for each dev's cell data (40-50 GB/month/dev over). The upside is that devs are effectively airgapped from the company, which I assume is the primary goal of all this.

Also, chat is banned. There have been efforts in the past to bring in tools like Slack, but the honchos believe that it makes the company better as a whole if we speak to each other like human beings, especially cross-department. Even if that means slowing down a project. And even if that means having to walk across campus, or occasionally driving to another part of town to another building. Phones are largely only used when off-site, or we need to talk to someone in another city.

It all sounds tremendously inefficient, but I try to think of it as being like working at IBM or Sperry in the 60's. It seems to work. The company is profitable and expanding, and has been for 40+ years.

Oh wow - I would not want to work there ever. I understand the need for some security but this seems just over the top.

They still inject that message now. I hit 90% almost monthly.

Wow I’m surprised. That is such a low barrier to doing your own BGP hijackig.

You have no idea

It's even easier to steal a phone number.

Lots of phone companies still just approve a port if you send them the required paperwork to initiate a port. That means with zero verification from the account holder a number can vanish from your account.

PacketCable (VoIP over Cable internet) is even worse

When it came out. If you wanted to "borrow" someone's phone number. All you had to do was clone the MAC address of the VoIP (EMTA) port

If someone called the number. Both you and the victims phones would ring

Same with CDMA. Just had to copy the phone number and ESN. Boom whichever phone was closest to the tower would ring, if both phones wear pinging with near freq then both could receive the same SMS and call, but only one could be on a call at a time. No actual interception.

Things got a bit different with MDN and MIN were different to ESN pair. Calls still came but you couldn't auth or call out for data services.

It's all a bit old now, but look up QPST, QXDM for the past decade and 20 years ago look up Oki900.

For some reason I think CDMA could suffer from crosstalk in certain circumstances, but it was something relatively obscure, like one device had to be above another (like an apartment building), but otherwise in the same/similar coordinates, and then certain code assignments would allow crosstalk to happen. It was weird, because it could be one direction only.

Unfortunately I don't really remember the details, since I worked on the core data network at the time.

Worse. Some very large carriers don't even look at the supporting documentation (bill, LOA) submitted with port orders unless there's a rejection from the losing carrier and they want to double check the address entered or something. Hijacking numbers is crazy simple. Same for hijacking the SMS functionality of any number in the US (voice traffic remains untouched). In about 10 minutes you can start receiving SMS directed towards any number you want, and also be able to send out texts originating from that number. Anyone who relies on SMS for any type of authentication should stop.

SS7 was designed in an era of huge Monopoly telecoms that all trusted each other. Worked fine. Needs to be burnt to the ground, the ashes stomped around on a bit, and rebuilt with the same level of thought that has gone into the development of TLS1.3 for modern use. Won't happen though due to the sheerly massive installed base of telecom gear worldwide.

Yes, sunk costs mean that while those costs are amortized these technologies will remain in place. Upside is there's plenty of business in the area of plugging the holes in the meantime!

> Anyone who relies on SMS for any type of authentication should stop

Err. That's pretty much every implementation of 2FA around the world.

Why isn't this more well known ?

Not nearly as many people know how terrible ss7 is and the lack of security/pki/crypto in old-school traditional telecom. It is also a lot more opaque to learn and has higher barriers to entry, and is a very clique like club of "Telco" people.

This is why the recent NIST guidelines on 2FA explicitly discourage using SMS. (Search for ‘SMS’ in the document: https://pages.nist.gov/800-63-3/sp800-63b.html)

The beauty of it is cases like Google's. They have this bizarre 2FA security-theater Google Authenticator thing, but then nearly force everyone to have their phonenumber as a "backup device".

Guess what the send you when you forget your 2FA or password? Yep, an SMS. So out the door goes the whole point of 2FA. Your three factors (account name / email address + password + Google Authenticator) have now been reduced to one factor: your email address.

I can rent a mobile tower in Malaysia or some other asian country, advertise your phonenumber as roaming there for about €10/h and start intercepting all your shit. Or just get your telco's inept service dept to forward your number somewhere else.

Lessons here:

1. Even the giants get it wrong. 2. There is no security anywhere in the tech world. Literally everything is broken. Your electronic car locks / starter system, your phone, your internet, everything is horribly horribly horribly broken beyond any imagining, even for hyper-tech savvy people. 3. Remove your phonenumber as a backup device from your google account and never use it as a backup device every again.

https://medium.com/message/everything-is-broken-81e5f33a24e1

Once you add another factor you can remove SMS from your Google account. I’ve done it with all of mine.

Edit: Oh, you said that.

I just removed my SMS from Google auth, thanks! And set up an Authenticator (Azure). I would like to see a world where we start removing SMS (and old passwords) from existing accounts.

> Anyone who relies on SMS for any type of authentication should stop.

Since the problem is that hijacking numbers is easy, shouldn't that apply to “anyone who relies on telephone numbers”, not just “anyone who relies on SMS”?

SMS isn't the only telephobe-number-based second-factor.

As someone who has tried to port about 1,100 phone numbers last year, this couldn't be further from the truth. Getting numbers ported is a GIGANTIC pain in the ass with ANY small detail being wrong will reject the entire batch. Since the slamming problem in the 90s it's been increasingly hard to port phone numbers without every T crossed and every I dotted exactly right.

Any sane bgp neighbor has prefix limit ACLs and prefix filters in place on their edge connections to another peer. As an ISP that announces a lot of space and customer space to 2 upstream transits, every time we take on a new downstream customer that brings their own /24 or bigger, we need to have our upstream transit providers update their prefix-list filters.

If our upstreams were clueless or negligent, it would be possible to get into a situation such as when a Pakistani telecom announced a huge chunk of V4 space that is YouTube, effectively DDoSing their international submarine links and also taking down YouTube for some users worldwide.

Except many of the big upstream providers don't actually filter anything. HE.net for example, doesn't.

We forgot to create some IRR entries and GTT just accepted our prefixes.

There is essentially no security, it's fairly trivial to hijack whatever space you want. (Doing it undetected is more difficult though!)

Well, not really. It doesn't mean that Comcast was re-advertising it into BGP and also advertising it to their peers. Usually you use a combination of prefix-list filtering and route tagging so you don't advertise garbage to your peers. RIP and other IGP protocols do not need to have their neighbors explicitly configured, they will find neighbors automatically on enabled interfaces. On some platforms they are enabled on all interfaces by default so it will form neighbor relationships with anything connected. A sane network design would be to only enable it on interfaces connected to your own routers (or customer interface, but on a different process ID) and also use a password.

Even for regular IPv4s you often get upwards of 20-40k SSH probes per day trying common passwords against root. IPv6 largely makes this go away since it's too big to brute force scan.

Yes, but the volume in kbps is tiny.

> Even for regular IPv4s you often get upwards of 20-40k SSH probes per day trying common passwords against root

So...anyone here ever set up a throwaway machine with root ssh enabled with one of those common passwords, so that some of those could get in, so you could see what they actually try to do once they are in?

If so, what did you see?

Yes, this is a pretty common practice. Take a look at the Honeypot wikipedia page and enjoy the rabbit hole. :)

Yes. And a lot of script kiddies (people who were just using easily acquired scripts to attempt to break into hosts). Looking at the sequence of password hashes you could tell some of these scripts were just fed the 'most popular password' list and were working their way down the list.

This is called a Honeypot.

https://www.youtube.com/results?search_query=ssh+honeypot

Reply on news.ycombinator.com