150,000 IoT Devices Behind the 1Tbps DDoS Attack on OVH | Hacker News Reader

150,000 IoT Devices Behind the 1Tbps DDoS Attack on OVH

securityaffairs.co

230 pointssengork114 comments

Only 150,000 ?

We see upwards of 2 million unique ipv4 sources scan us on port 23 every day. These are all compromised IoT devices and routers.

In the past hour we saw 350k+ unique sources.

In just the past 3 minutes that number is 168,230

Top sources in the past 3 minutes:

  848 211.201.69.50
  840 180.66.99.72
  838 222.121.157.61
  759 95.17.97.136
  639 171.248.123.112
  542 189.78.49.194
  511 176.109.222.124
  386 60.249.84.179
  378 118.161.69.18
  377 61.75.42.129
  252 125.142.55.218
  252 183.102.221.85
  245 106.186.20.183
  233 112.162.191.217
  203 121.143.65.181
  199 115.86.134.94
  190 89.163.242.12
  183 91.205.123.37
  181 86.90.10.151
  179 91.240.140.14
  177 191.103.72.251
  173 185.129.2.236
  169 218.201.74.122
  168 116.99.113.72
  164 82.119.65.190
  160 118.129.105.9
  158 194.88.205.101
  156 77.88.202.60
  156 82.79.75.5
  155 112.165.227.205

We see 2000pps of this shit all day every day. No one cares.

Is there a database of this so people can check if their devices are compromised?

I could publish it somewhere.. The problem is people just don't care.

I dumped the ASN owners for a bunch of sources, the top 3 are:

  15664 BR TELEFNICA BRASIL S.A, BR
  15215 VN VNPT-AS-VN VNPT Corp, VN
  15112 CN CHINANET-BACKBONE No.31,Jin-rong Street, CN

You think TELEFNICA BRASIL gives a shit that they have 15k+ compromised customers?

I'd say if you can telnet to your wan IP and get a login prompt, you have a shitty router that exposes telnet to the world and you're probably compromised :-)

You think TELEFNICA BRASIL gives a shit that they have 15k+ compromised customers?

Shouldn't they? It eats a lot of their outbound bandwidth. Even if they have peering agreements that they don't have to pay for, there's still the cost of the equipment and their internal network bandwidth to consider.

Some kind of IoT realtime block list like we have for spammers would be nice

It's ultimately not very useful for that sort of thing.. 326,160 addresses in the past 38 minutes. What would you do with this information?

Chrome, Firefox, IE, could all check your IP against that list, and inform you, "Looks like you have an unsecured device which is attacking the internet, and slowing down your connection. Here's a site that can help you fix the problem..."

I suppose, but that would require you to install the add-on that does that, or for it to be included by default.. both of which seem unlikely.

Having google or facebook check REMOTE_ADDR for every client would work though, but most people wouldn't understand what the notification means. Plus, on a network that isn't using nat you'd never get the alert since the IoT device itself would never be REMOTE_ADDR.

True, but it still would be a step in the right direction for home users.

shodan.io

Can you please explain or provide a reference link to provide more explanation:

>We see upwards of 2 million unique ipv4 sources scan us on port 23 every day. These are all compromised IoT devices and routers.

What is the unique connection between port 23 and IoT devices? Genuinely Curious.

Thanks.

One of the observations I read in the aftermath of the Krebsonsecurity DDOS was that the vulnerable IOT devices often permit telnet (port 23 and non-encrypted) connections. A common approach to take over one of these devices starts with brute-forcing the login prompt.

Once compromised and running malware, the device proceeds to probe random IP addresses for open telnet ports to spread further. The uptick in port 23 connection attempts has coincided with the perceived spread of IOT malware.

I wonder if someone should grey hat this and scan and kill vulnerable hardware.

Once one of these botnets finds a device with admin:admin credentials the first thing it can do is change the password.

Shitty underpowered devices that ship with telnet enabled instead of ssh.

We run a distributed network of PoPs in the cloud and get easily many times this, as well as massive ssh brute force attempts and other more bizarre forms of scan/attack. I sometimes look up the IPs and check them out and they are ancient cameras, routers, and things like Windows 2000 Server instances sitting on the Internet running malware.

Can you share what is the methodology you are using for determining the source IP is an IoT device? Are you just nmap'ing them and using OS/TCP finger printing?

I had the same question for how OVH was determining that X amount were IPs cameras vs X amount were DVRs etc.

It's just what they are. Routers and IoT devices exposed with telnet running and default credentials.

It's probably mostly routers, but the botnet doesn't care. If they can connect to you, send root\nroot or admin\nadmin and get a shell, they will.

OK I see, so the router the IoT devices are in back of as well as the IoT device have both been compromised. And the owners of the botnet have made sure that these IoT devices have proper NAT/PAT mappings on these routers.

No.. separate incidents. People often forward ports to IoT devices or stick them on public interfaces in the first place.

How do you tell the difference between a router and some device sitting behind the router?

Some idiot has been attacking me all month:

127.0.0.1

I keep counter-attacking him, but he's always one step ahead of me!

I'd try SSHing into them - maybe they have the same login credentials as you?

I got a kick out of this, thanks!

A lot of people including numerous EU regulators would think that posting such a list of IPs is very tacky. At Google I would be reprimanded and probably fired just for producing a list like this, or even writing a tool that was capable of producing a list of non-anonymized IPs of clients.

Now that I got that out of my system: where's all the v6 traffic?

Those are not "clients". They are random hosts on the internet trying to compromise hosts on our network.

Even a light bulb has inalienable rights in the EU, comrade.

As system administrator of my home network, it worries me that a device on my network might be involved in an attack like this, and I would never know.

Maybe the target of such an attack could gather a list of IP addresses used in the attack, then pass them to Google, who might warn on their search homepage if you browse from one of the IPs on the list? (e.g. "Some of your internet devices may be at risk, click here to find out more") I know IP addresses are a poor proxy for identity, but it could be a step in the right direction.

Google already has a similar mechanism in place where they require captchas from IP addresses that abused Google APIs previously.

The combination of that with Google Shield might actually work to inform the users, but then again users are confronted with similar warnings from abusive ad networks all the time, and probably learned to click them away fast and forget about it.

Simply set up your firewall to drop outgoing packets with source address not belonging to your subnet. The DDOS slaves are usually sending packets with spoofed source ip addresses

This is usually only true for amplification attacks. The new 1Tbps and 600+Gbps attacks reported about are direct traffic attacks from the actual bot IP addresses.

Terrifying that you can get that kind of volume without amplification.

Not to mention that this is 1-30MB/host. Just wait until gigabit for consumers is the mainstream.

> The DDOS slaves are usually sending packets with spoofed source ip addresses

I don't understand why this works. Why doesn't my ISP simply block outgoing packages with "fake" source IPs?

Most of them do that already. It's called BCP38. ~20-30% of ISPs however don't.

To quote Krebs [1] on that subject:

> BCP38 is designed to filter such spoofed traffic, so that it never even traverses the network of an ISP that’s adopted the anti-spoofing measures. However, there are non-trivial economic reasons that many ISPs fail to adopt this best practice. This blog post [2] from the Internet Society does a good job of explaining why many ISPs ultimately decide not to implement BCP38.

[1] https://krebsonsecurity.com/2016/09/the-democratization-of-c...

[2] http://www.internetsociety.org/deploy360/blog/2014/07/anti-s...

I read the article. The reasons are trivial.

First, old (>10 years) networking hardware may be unable to support it. All new hardware can do it, but some old stuff can't, and some ISPs haven't budgeted for the update. Response: 10 years is forever in the hardware cycle. This isn't a woodworking business, where old heavy iron is a good thing. Sensible businesses budget for returns on investment and mean time between failure on shorter time scales.

Second, the labor to install network hardware replacements and perform configuration updates is expensive. Response: That's literally your job, you don't get paid to sit around and collect money.

Third, and most importantly, the costs of the DDOS are not felt by the ISP. It's a tragedy of the commons. Response: Regulation, obviously, is required. If your network causes damage that the industry says you should have prevented, you should pay.

ISP > you don't get paid to sit around and collect money.

I don't think you understand the purposes of ISPs in the US :)

Maybe some of the larger providers should stop pairing with these networks? Clean up their acts or have no business

The largest provider in my nation is one of these networks, they don't do a thing until forced my the regulatory body.

And anybody can test used ISP to see is spoofing allowed or not: https://www.caida.org/projects/spoofer/ This 20-30% should be first named (and then pressured to implement BCP38, but it is hard to do).

Fake where, and block where?

If you fake your neighbors IP address then the hacked IoT device never gets taken down, and innocents get bothered unless the ISP does a good job investigating.

Will the ISPs monitor faked packets inside their own network? And if they do at what level? Individual modems, entire segements? Entire regions?

Pure laziness.

No home network or consumer ISP should allow spoofed source packets beyond the edge. The spoofing problems come from cheap / shady dedicated + VPS providers.

Correct. I use

    # deny spoofers
    extinput="iptables -t filter -A INPUT -i wan"
    $extinput -s 127.0.0.0/8				-j REJECT
    $extinput -s 10.0.0.0/8				-j REJECT
    $extinput -s 172.16.0.0/12				-j REJECT
    $extinput -s 192.168.0.0/16				-j REJECT

There are home network monitoring products. Here's a roundup: http://thenextweb.com/insider/2016/01/04/4-devices-that-can-...

Anyone know of a good open source solution that doesn't require a huge amount of fiddling?

Ironically enough, this is a place where I think that remotely managed routers (like the OnHub) might be a fantastic solution for the average Joe. Dynamically identify problematic traffic patterns, and block them at the home router.

It would be an 80% solution at best, and I wouldn't want one; but I trust Google more than I do my mom when it comes to managing the packets coming out of her network.

@internetofshit will have a field day with this.

In all seriousness, this is only going to become worse in the future. Can't wait until the day when smart fridges, toasters and bicycle locks join in on a multi-Tbps attack and break the entire internet.

Internet providers will curb it by reverse-firewalling all consumer connections, maybe with support from the copyright abuse lobby, so as to kill P2P and any other advanced internet usage for that matter.

The unfortunate and dangerous thing here is that nobody really has an incentive to fix this. The Internet doesn't belong to anyone.

This really needs to be fixed at the national or international (IANA?) level by mandating the deployment of anti-DDOS source quench and anti-spoofing measures. Any owner of an IP block should be able to register a key and then send source quench messages and this needs to be deployed uniformly.

But I'm not holding my breath. It's like herding cats, and as a general rule nobody anywhere cares about security unless their house is on fire (and then they go back to not caring after the fire is out).

Another thing that needs to be done is to cut off support for this activity. It needs to be made illegal to pay ransom for DDOS or ransomware for that matter. If you get ransomwared or DDOSed that sucks, but that doesn't mean you should be allowed to reward the behavior and finance it being done to others.

So if a hospital has vital patient information locked up with ransom ware, you want to make it illegal for them to pay to get it out?

Your source quench nonsense sounds like just another piece of pointless infrastructure to be abused.

You can't have "Intelligent" without "Intelligent(ly sending over personal data to servers you don't own for fun and profit)" in your devices.

Because connecting your toaster and fridge over a local network, and running updates via your PC/laptop would be just too complex

I wonder if at some point a government or two will just decide that the dubious benefit of a wired fridge isn't worth the national security risk of increasing existing botnets by an order of magnitude?

If it breaks the internet, the perpetrators will have no way to shut it off. Internet armageddon.

It's unfortunately way too easy to find such devices. A quick scan of the (less scary) end of the ipv4 address space and I was able to find ~15k cameras and I was only searching for a couple of models for fun... Here was the result: http://opencam.ma.rtin.so/ -- most of the pins probably wont work anymore, as it's a couple of years old.. Still crazy.

I was able to find a few still working - in England: leamington spa, birmingham, Bradford (appears to be covered in cobweb!) Carlisle, etc.

EDIT: I'm going to be spending hours on this today!

Heh, it looks like ShodanHQ now has a section entirely dedicated to this... https://www.shodan.io/search?query=Server%3A+SQ-WEBCAM -- using their API, and a geoip database (eg: maxmind) you could build something similar in a couple of hours.

You should also be aware that you just admitted to breaking the law in most countries.

Edit: I take that back. I assumed "open" meant a default or insecure password

Wait, aren't these public?

Unfortunately in the hacking world, "access" doesn't mean legal access. Even, if it's there on a webserver with no login/password and directly on /index.html

It's a completely fucked situation we're in, with the CFAA law. It allows the feds to charge anyone they like, cause they used a network.

I guess we're supposed to fax the owners before we submit a TCP connection with their machines, but we'd probably run aground of fax spam laws.....

Jesus I was just thinking about the consequences of no patch routine in the IoT device world. And, here it is. :)

Imagine having to internationally co-ordinate patching of 150000 devices. Because the alternative is that 150000 homes will have their NATed IP-addresses blocked from each service being attacked.

Just wow...

That would mean blocking the ISP, and every ISP, so it's the end of blocking because there won't be any legitimate traffic left.

The manufacturers must be both sued for selling exploitable devices and educated about how to write secure software.

There is another post on the home page of HN about the security of the Linux kernel https://news.ycombinator.com/item?id=12589894 That's very important for this kind of issues because many of those devices are probably running on some Linux distribution.

This is why a proper and solid patch routine is of utmost importance.

I'm a devops guy so I'm basically a sysadmin and I've been an advocate for patch routines for many years now. In a climate where people are almost offended when you tell them they need to patch their servers regularly.

So if IoT is really the Internet of things and not the Intranet of things, then they need solid routines for patching their software.

Yeah, and having an automatic, secure, reliable patching routine for embedded devices is not trivial (and thus really expensive). What's even worse, we already have millions of deployed devices, and some of these probably don't support upgrading the firmware at all, or require manual intervention from users who have no idea they are part of the botnet.

> Yeah, and having an automatic, secure, reliable patching routine for embedded devices is not trivial (and thus really expensive).

Roku (and others) seems to have figured that out. They cryptographically sign each update.

Roku legitimately needs to initiate connections outside the home. Can the same be said for e.g. a light bulb? If switches don't drop these packets, then routers should, and if they don't, then ISPs should. Is there a field in DHCP that could be used to communicate the fact that a particular host should generate no outside traffic?

Don't assign the device a default gateway (or set it to 0.0.0.0 if one is "required").

I have thought for a long time that we'll one day get to the point where the best thing one can do for a host's security is to not allow it to generate traffic that can reach the Internet.

Just like we have default deny on incoming traffic, we need to start using a default deny on outgoing traffic as well.

My first thought would be to set the default gateway to 127.0.0.1. It should mean that they can't route packets to anything outside their LAN?

I meant that the light bulb could tell the router "I'm IoT so assume I'm pretty dumb", to which information the router could respond in any number of ways. I don't think your setting would have the effect we want, however. The light bulbs have to talk to whatever is supposed to control them, so they have to be able to see the LAN.

The gateway is needed to route outside of the local subnet, so if the bulb is 192.168.1.17, it can talk to anything in 192.168.1.0/24 (presuming a standard home user setup), but anything else would get 'no route to host' errors on initial connection attempts.

You'd need to configure the DHCP to hand out these kinds of leases by MAC address though, as I can't see vendors agreeing on a way to easily restrict the devices net access! :-/

Perhaps your average router needs a button to 'add device', only allowing new devices access via something like the WPS button with a term second window for new DHCP request incoming? Otherwise the DHCP ignores any incoming request, just sleeps. Adds one step in the quickstart guide.

>so it's the end of blocking

not really, ISP gets dropped, goes into panic mode and FINALLY fixes its own shit, finds the endpoint originating illegitimate traffic and blocks it, reports back upstream about the fix, gets reinstated, finally starts monitoring for ip spoofing.

Another problem: video service A runs on server farm X which blocks IPs of ISPs known for being the origin of DOSes. Video service B runs on server farm Y which doesn't. Who makes more money between the two video services and between the two server farms? As long as Y is not hit, I bet on B and Y.

Furthermore, but I'm a noob at this, isn't the upstream Internet provider that should block traffic instead of the server farm? Otherwise it would get all the load anyway. Am I wrong? (probably yes)

Just filter the BGP AS of the offending ISP. That's how it should be done. Block the whole ISP at AS level. Maybe automated time based threshold filters w/ distributed banlist. Just like fail2ban but for BGP and with a distributed banlist (AS + timestamp of exceeded threshold).

In many cases there is no vulnerability. The box has default credentials of admin:admin and someone puts it on the internet and never changes the password.

you dont have to coordinate any patching, just null route end users who are the source of DOS and let them deal with it. Its them who opted for $25 with free shippintg 720p 'Cloud based, works with your iphone' nanny cam.

http://blog.level3.com/security/attack-of-things/

Getting manufacturers to patch, and users to update these embedded linux devices is going to be pretty hard

Then just hijack them and set the default gateway to 127.0.0.1. Problem solved.

And now let's apply such a scenario to autonomous vehicles, on land and in air.

but rather than causing a virtual DDOS, now in physical space. shutting down a whole city, for the lulz.

IoT and AV show that the "Facebook" method of software development - move fast, break things, agile/scrum, whatever label is used for non-engineering, will not work for the next stage.

ditto the skills of most young CS grads. most companies can't even secure their shitty email services - but cars is easier?

a whole new supply chain for code needs to be developed, from languages to curriculums. take what the airline industry has been doing and commoditize it, it must be braindead easy to build a secure and robust piece of code for this new world.

I remember when the ntp exploit came out few years ago datacenter where we have a rack contacted me saying the Supermicro IPMI devices on the Supermicro servers were participating in an amplification attack.

I was like wtf! Matter was quickly resolved of course, also they learned a lesson and moved ipmi ips to 10mbit limited connnections not 1gbit.

Tho ideally a local ip that accessible only via a vpn would have been the best option for remote management but yeh, little steps I suppose with some providers.

The problem is that that there are ISP's who are not implementing BCP38 (http://www.bcp38.info)

This will only take us so far. Many of these botnets are large enough that they don't need any spoofing/amplification to be effective.

How does that stop spoofing addresses inside the segment?

If I am 8.8.8.8 and I fake 8.8.4.4, you still get my traffic, and someone else gets the complaint.

If you are 8.8.8.8 and you fake 8.8.4.4, it's likely the same person getting the complaint.

If you just picked those two addresses at random from the entire range just to serve as an illustration, you should have bought a lottery ticket instead.

It won't stop spoofing, but if you assume that the botnet hosts are evenly distributed across the address space, you're going to cut the spoofed traffic to 1/(256^2). At that point, you might not even notice the DDOS.

we need customers to want BCP38 implementing routers in their homes

But... the routers and IoT are the ones often compromised. The ISP needs to enforce it.

Also, anyone who understands networking (everyone on HN, for this purpose) should have a default-deny firewall for at least their IoT devices, if not every device on their network.

how many years until iot manufacturers get sued for producing insecure devices?

I think the reason we have such bad security is that non-technical end users just don't care. You can try a dozen different approaches to getting them to care about security, but they often cannot be bothered with it.

Thus, if you want to go the legal/penalty route, you need to sue the end users. The entity that owns the house/office that installed the unpatched CCTV camera is effectively responsible for the behavior of that camera. If they then want to shift the responsibility to the manufacturer, that's their choice (and effort).

What it will do is make users consider a bit more carefully when choosing devices and manufacturers, and it will make manufacturers have to consider (and promote) their security and patching practices to maintain marketshare.

I can't see how end users could be possibly empowered to analyze the IT security risks of off-the-shelf IoT devices, before buying them.

There are already safety standards and according mandatory certification processes in place that (should) prevent electric appliances from burning down your house (CE) or from bringing down airplanes (FCC).

What is (urgently) needed is a similar approach to mandatory IT security certification for IoT devices. This is also advocated by Bruce Schneier [0]:

Security engineers are working on technologies that can mitigate much of this risk, but many solutions won't be deployed without government involvement. This is not something that the market can solve. Like data privacy, the risks and solutions are too technical for most people and organizations to understand; companies are motivated to hide the insecurity of their own systems from their customers, their users, and the public [...]

[0] https://www.schneier.com/blog/archives/2016/07/real-world_se...

I can't see how end users could be possibly empowered to analyze the IT security risks of off-the-shelf IoT devices, before buying them.

The same way they do for everything else. Ask experts. Demand third-party reviews. Require warranties for security flaws.

Want an answer? The real one? Let corporations attack each other via electronic attack.

It will be far worse at the beginning. But once you really threaten the bottom line for free, security will come.

I sadly see no other good way to have security in software.

http://www.ul.com/newsroom/pressreleases/ul-launches-cyberse...

Suing users wont work, now suing the manufacturer if the device is possibly a path that we will see pursued.

If a precedent is set (after all cars get recalled all time for flaws) then we would endup with a more secure internet.

In the same way we sue users for purchasing cars that get recalled, Samsung Galaxy 7 Note, etc etc?

This is not a user issue. Manufacturers need to issue recalls and be made liable for shipping insecure by default products.

Good luck finding that vendor on Ali-express or Amazon.

Not gonna happen. Think of all the insecure windows "devices" out there. How do you draw the line.

You draw the line by legally kicking the crap out of a smaller vendor, then they become an example for the bigger guys. Suddenly the industry-wide Cost:Benefit analysis changes such that it is less costly to secure the devices they sell.

It's not all black/white. Nobody is immune to this problem, even with best efforts/practice, saying "let's sue to death small vendors so that big one react" is kind of ridiculous. Again, it's hard to draw a line.

Finally, the WiFi router invasion that we were warned of as early as 2007 is coming: https://www.flickr.com/photos/dullhunk/3109815261 (original source from 2007 is 404)

There needs to be an international standard to avoid things like this.

It should be easier managing devices that have access to the Internet on the router level.

Most can't understand access restrictions, IP Tables or installing custom firmware. There needs to be a common standard, API on each router to manage devices connecting to the Internet and seeing which devices do and don't.

This would open the doors to creating apps etc and possibly help mitigate threats from unknown Chinese IoT devices.

I manage a huge fleet of raspberry pi in my jobs. There are geographically everywhere.

I wish that there will not be found by some bad guy, but I know our system and I'm 100% sure that will happen one day. We have a basic level security, like so many other startup in that field though.

And you do nothing? Because of you many business could loose their reputation and money.

At last, a legitimate reason to use a green-bean HN handle.

Feels like how things might have been when home electricity was first becoming pervasive.

Lots of dubious devices and a laisez-faire approach to eg. electrocution risks and fire hazards.

After enough public outcry regulation is introduced, standards are developed and enforced and your television is no longer at risk of bursting into flames or frying the cat.

Or, in today's world, of being conscripted into a global botnet and DDOS'ing your neighbours.

This isn't bad enough, not yet, for some kind of protocol that allows source quench / notify a remote ISP of a suspected infected host and suppress traffic from said host.

It would need to be out of band, and I suggest it use OpenPGP for signatures (chain of trust from IP allocating bodies), actually it would also need to query a database of allocated IP ranges.

IoT = Internet of Targets

Something needs to be done about DDOS at the backbone and tier-1 level of the Internet or we are going to lose the public Internet.

the problem is more complicated than you make it out to be. DDoS aren't always some special kind of packet or traffic that is easy to identify, it is just a flood of normal looking traffic from a ton of compromised sources

I didn't say it was easy. I just have the impression that nobody is even really trying to do anything about it Internet-wide. Instead we just have protection as a service (Cloudflare, etc.) which provide protection but do not actually solve the problem. (... and I wonder if these DDOSes are going to get big enough to take them down eventually?)

Source quench would help. It's not a silver bullet but it would make this a lot harder. In many cases (including ours) there are ways to ID legitimate traffic vs. junk.

Any idea why OVH is being targeted? What customer is OVH hosting that they're trying to hit so hard?

I wonder if you use the DVRs for bitcoin mining how much you could produce per day.

Most IoT devices wouldn't produce any significant revenues from Bitcoin Mining because they don't have anywhere near the computational power of dedicated Bitcoin mining devices.

A DVR might have a chance to make an impact if it had a GPU that was used to encode video that could be co-opted to mine Bitcoin, but I think most DVR's use special video encoding chips rather than a general purpose GPU.

I think the only hardware they would have that could mine Bitcoin would be their general purpose CPU, which is probably under-powered anyway.

A DVR would be more easily monitized as a node in a Botnet that does DDoS attacks, email spamming, or network scanning.

One resource they do have though is drive space. a DVR botnet could sell unused DVR HD space using a service like Maidsafe [1]

[1] https://maidsafe.net

How are they able to identify that these were IoT devices?

Reply on news.ycombinator.com