Potential security flaw in network (digitalocean.com) | Hacker News Reader

Potential security flaw in network (digitalocean.com)

17 pointsjboger14 comments

Digitalocean.com has misconfigured their network in a way that allows for anyone to monitor customer network traffic. During the process of checking MySQL replication between master and slave, I noticed there was a lot of background noise in tcpdump. It seems DigitalOcean has, using KVM and libvirt per their own recognition, put the libvirt-interface in an overly large bridge, and then kept applying more and more networks (multiple /24, it seems). While this might be a convenient way of assigning new networks to an ever-growing customer stock, it also sort of turns the entire thing into an amateur radio station (using the word amateur here to denote the activity, not the skill level of Digitalocean staff!). I do not want to be able to read what goes on with various mail, ircd, web and Microsoft sql servers, in networks far outside of my logical reach, as a customer with one IPv4. I am not an angry ex-customer. I will keep using their services, if this is fixed. I will not paste logs as that would add nothing to my disclosure, more than a possibility to exploit innocent users. I wish to encourage the community to take a few steps back and not engage in target practice, while Digitalocean undoubtedly remedies this situation (I have been in contact with them repeatedly before coming here). I hope that this helps, for whatever it's worth. This is where my involvement ends. I leave this information in the hands of the community.

Best Regards, Johan Boger (also posted on full disclosure).

I'm very surprised by this. I see (or assume I see) the server you used to have. Nor can I find a ticket related to this issue. I don't want to flat out say "Will not fix -- can not reproduce" but on the other hand, I can't fix it if we can't reproduce it.

Where do you want me to email logs?

Thank you for your reply! I will send you user-id, so you can check tickets, and I will email you a full tcp-dump from my vps.

Best Regards,

Johan Boger

I have now talked to Digitalocean, the issue has been identified and a fix has been applied.

Fast, courteous, responsive. There is something to learn here for other cloud shops.

I noticed something similar to this (not sure if it's the exact same) - posted it to HN here: https://news.ycombinator.com/item?id=5734960. Not sure how far DO looked into it, but it was considered normal behaviour at that time...

I'm not seeing this on my DO installation.

Although, I did notice that my installation came setup using Google's free public DNS servers (8.8.8.8, 8.8.4.4) -- curious that digital ocean isn't running their own DNS servers. If google has an outage, thats a lot of customers who will be complaining to Digital Ocean when they have 0 control over it.

I can confirm it is still displaying sensitive information.

Have you raised this with digital ocean? That should be your first port of call before you go blurting it out on the internet.

As per guidelines of responsible disclosure, I contacted digitaloceans a good 10 hours before going public. I also contacted them again -before- going public, telling them I would do so, and why.

I feel I have done my best to ensure this does not harm them in any way, while making sure customers are protected.

I also want to point out that this entry was truncated. My full disclosure is, suitably, on full disclosure (the list). There, I explain that I consider DO one of the best cloud shops out there, and I will happily continue using them. I am not an angry customer. I am a current customer, but this is still information that people should be aware of, I believe. Anwyay, do with it as you please. Digital Ocean are aware of it and on top of it (I believe). I wanted to give customers the same chance.

I think I am. Can someone verify that I'm doing this correctly? I'm running "tcpdump not host [my ip]" and all sorts of packets that aren't related to my VM are showing up. Doesn't seem to be full connection streams, though, just a leaky packet here or there.

Example: 17:27:43.293749 IP [not me] > [also not me]: Flags [S], seq 324665288, win 65535, options [mss 1460,nop,nop,sackOK], length 0

I'm in SFO if it matters.

It was, indeed, a matter of leaked packages intended for another recipient, but sent to all when not received properly. You are doing it correctly, but it should as of today no longer work. Can you try again?

jb.

Yep, looks like it's fixed now.

Indeed. I believe it might still be possible to arp spoof, if the arp table is leaking MAC. There seem to be no more IP packets leaking in this manner, though, which was the real concern, imo. DO has probably already remedied the rest, and if not, they most likely will shortly.

All in all, a productive communication that worked out well.

Thanks, guys!

Reply on news.ycombinator.com