kitteh

There's another forum where a Jacksonville ARTCC controller echoed they had 15 people call out (not due to some anti vax movement) and had weather looming on top of the military who takes over half their airspace and doesn't give it back promptly (or the FAA gets the word but doesn't pass it along). Combined with constant understaffing and the fact that JAX can't stop departures from places like Atlanta (but other centers can) doesn't help.

It should be. Also this was covered in a reinvent presentation about the aws network.

Inter region traffic always goes over the backbone (this includes EIP to EIP). This also includes going from EC2 to any service like S3 in another region.

Except China. China to rest of world is not via backbone.

It would revert to not being signed, which routes just fine. You just don't get the additional security benefits. It won't turn it into invalid if I'm following what you are saying.

There can be legitimate use cases why a network maybe have a very few amount of prefixes not signed or even invalid: canaries and beacons.

For example, running tests to a signed, unsigned and invalid prefix can provide insight into how other networks are routing to them.

One example is a beacon to probe to determine if a network has enabled origin validation. Failure to connect, or a change in the routing path can provide insight into which networks on the internet have enabled origin validation.

Lack of redundancy due to cost probably has something to do with it.

Same issue with internet peering in quite a few cities (no viable 2nd location, costs a lot of money to double up the gear, additona optical transport).

AT&T has been pushing really hard on folks in the last year to take early retirement packages that would give you a fraction of what you would have gotten if you stayed on until being eligible. Friend of mine recently took the package and found a job elsewhere and commented that morale is pretty down if you are in that camp.

I was a big MaxTNT customer. The cooling on those were interesting. Also worked with the earlier Cascade BSTDX 9000 and CBX 500.

The most brutal thing I saw was when someone compromised a customer ISDN router (the small Ascend boxes with the curses UI) and changed the creds to login to their ISP and disconnected it and forced it to redial repeatedly. The local telco charged you for every ISDN call if it was a business line and since ISDN call initiaton/setup are instant - they had a several thousand dollar phone bill. I recall seeing the RADIUS server getting slammed with auth failure for days when that happened.

Tragically there was nothing simple about ISDN. A customer would have to know their switch type at the CO and there were a number of other things that could break it. At least with a T1/DS1 you only had to worry about SF/ESF, B8ZS/AMI and number of channels.

Yep it's the main spot in nashville for voice stuff. Has local access switches (dms 100), tandem, and LD goes out of there. At&t I think also has their long haul optical transport and some core ip backbone stuff there.

Back when I worked in a CO the engineering goal was to be able to run the building load on gen for at least 24 hours straight before getting fuel trucks. We had weekly gen tests to validate things were working as expected.

It could be that they had to de-energizd some equipment to perform inspection and work within the facility. I know our facility had a number of procedures on how to de-energize parts of the building and inhibit the generators from feeding that area (a lesson learned in the Hinsdale CO fire years ago).

During Hurricane Katrina a number of us had to invent a procedure for de-energizing non critical equipment to reduce power load in order to keep critical services running for an extended period of time since it was clear we werent going to get utility and resupplies for awhile.

Dark docs is kind of watered down history. The previous mentioned book is great. Back in the 90s Discovery actually had documentaries that covered this from the people actually involved (including the person who proposed the operation) - now we have SHARK WEEK.

Cables don't equal good connectivity.

Interconnection facilities are the best places to be (or rather what you want to be connected to in order to establish as many adjacencies). 111 8th in NYC. 165 Halsey in Newark. 350 easy cermak in chicago. Palo alto for the bay (but also eqx in San Jose). Infomart in dfw. Etc.

Cable landing stations generally service one cable and the actual networks that use it are generally some distance away.

Yes. Which raises the question why don't we have something to detect/mitigate this.

Yes. Phone phreaking in the 90s involved a ton of learning the lingo and the ins and outs of the phone company. It was even weirder to go work there later on and found how much that paid off and earned trust with a lot of the older folks there.

Or ddosing a site offline.

Remember that only a handful of companies today have the infrastructure to handle large attacks.

They could use their own timing source, though? I recall back in the day being able to pull timing source off a circuit.

Chargen isn't that popular now.

The big things these days are:

spoofers who generate a ton of syns to legit destinations which result in a lot of syn+ack to the victim. Bcp38 would help here.

Botnets generating a ton of UDP to destinations. Hosters, cloud providers (especially those with vulnerable/open EMR clusters) and broadband ISPs with easily compromised customers are the problem here. Kudos to those who take down the botnet command and controls.

Memached/ntp/cldap amplifiers. Still out there, still a problem. Thankfully a few of these services are policed at large peering interconnection points.

Yes. As someone who has to support network infrastructure and had many network issues (at all levels) bubble up to them since the start of covid/wfh - broadband networks are being stressed during work hours. You can run your own monitoring at home to validate it to differentiate between last mile vs. peering related.

The challenge is some of thr broadband networks aren't being transparent about it. They won't admit to it, etc. Luckily I've found folks in a few companies who are willing to go off the record and explain what is the state of the last mile and give dates around planned upgrades (which are impacted due to permitting delays due to city employees struggling with covid restrictions).

I highly recommend getting two ISPs at home (cable + DSL/fttx/etc.) if you can.

As for Corp networks...they should be doing more with providing their employees some tooling to indicate if the issue is their ISP or the Corp network. It's a sad state that VPN appliances cost as much as they do (surprise - you could probably build a cheaper/better one but something something no one ever got fired for buying Cisco/etc).

I remember as a pc tech (before they did that geek squad stuff) having to validate computer returns. The amount of stuff people would pull daily always amazed me. Putting bricks in a computer box, an old video card for a matrix vga card, the most wild was some video equipment where they replaced it with a two liter bottle of urine. I remember confronting people once I opened it and they would always pull the "well my son packed it and gave it to me" and promptly leave (with their item). They usually worked multiple stores in the area and sometimes we'd chat with the other stores, but usually that was for the bigger scams (people using altered receipts, asking for cash refunds on multi thousand dollar purchases).

Could you drop some IP prefixes you believe are hijacked and timestamps? I can probably help out and bring this to folks who can take action.

(I have to deal with hijacks frequently and part of our investigation is beating on folks who have permissive filters and pressuring their peers to improve things)

Yes. Many of these shops follow the same design patterns. Ports for hosts at L2 that they bill on that are part of a big Vlan/L3/SVI interface that has tons of customers. I've seen these configs where folks have hundreds of "secondary" ip addresses setup where any customer can steal other customer IPs in addition to lack of anti spoofing. It's slop and it's tolerated.

It'd be nice if we could address some things like BCP38 (anti spoofing), RPKI, route filtering and folks who knowingly support infrastructure that's used for outbound ddos (c2s and regular hosts), spam and malware phishing. Plenty of hosting shops in US and Canada have these problems. That seems a bit more within our reach whereas an ISP in India is more than happy to pay a vendor to implement middlebox packet molesters.

They're already doing massive flow sampling at the GFW complexes today. They can police down to individual flows if they want to. They scale wide by distributing out traffic based upon src or dst IP.

Friend of mine in Toronto told me a story how a CRTC (canadian telcom/radio regulator aka like the US FCC) tech was walking around his apt bldg with a spectrum analyzer. Pin pointed an apartment down the hall and was able to make contact and remove the offending device that was bleeding into mobile frequencies. Was a Chinese manufactured baby monitor that was interfering with mobile service. Said they had this problem all over the country with shoddy imports.

There's a few around the world that have been essentially abandoned. High cycle time aircraft, engines stripped, avionics gone and useful parts taken.

There is a whole industry around salvaging old jet aircraft for parts, movie sets, furniture, etc.

An old 747 fuselage would be well under a million. Even less for 727s and 737-200s.

Back in the day we had GOCO (govt owned company operated) facilities to handle things like ammunition manufacture (they still exist). But this administration has made it clear they won't do that for PPE.

It was popular for booting people off IRC, but there were other exploits around the same era that did the same such as land and teardrop.

It wasn't super notable. What was more horrific was the amount of windows machines that had tcp ports for various windows services open to the internet that led to not only crashing but remote compromise and rootkits/botnet stuff. That went on for years and only got mitigated by people deploying routers with fw/Nat functionality.

I can relate and drove on the highways that mostly didn't have lights. Connecticut/NY to Montreal almost every weekend for a year (except in inclement weather), usually late at night.

Lots of great radio programs. This American Life, Art Bell, Hearts of Space. Hell going back far enough even Loveline had it's moments. I forget which university (in Burlington) had a great radio station that played good music and recorded every episode. Would get home and download the recordings and still listen to them (introduced me to a bunch of artists I didn't know).