Cache on Delivery (Mining random people's memcached)
slideshare.net
slideshare.net
* You need to make sure you've firewalled (and/or on single-host configurations bound it exclusively to lo0) memcached so people can't talk to it from the Internet.
* If you survey the Internet, you'll find a lot of people who have not obeyed this rule.
* If you don't obey this rule attackers can get and possibly alter most of your data; an exposed memcached is probably game-over for your site.
The only things you should be able to talk to on anything in your data center are ports 443 and 80, plus 25 on your single mail relay if you have one, plus 22 (heavily filtered) on a single relay SSH server. If you can talk to more than that, you've probably done something very wrong.
That alone allows you to avoid most attackers trying to guess valid server ips.
Now to put a honey pot on port 22.
I can't emphasize this one enough. Unless you need to login from a lot of different machines, there really isn't any excuse not to do this. It also has the bonus of making logins really easy since you don't have to type a password.
Also, yes, among with changing the port the only way that should be possible to get in is through 'keys.
Bad gowalla, bitly, and PBS operations departments. Shame on you.
Unfortunately it's often not good enough to point out open ports, and one needs to demonstrate exploitability. The main thrust of the talk was so say "if you come across memcacheds, don't skip them, there's coolness there". e.g. an open memcached used by Django directly equates to remote code exec due to Python's pickle.
You're spot on with the final point; firewall firewall firewall.
[fd: that's my name on the preso]
As far as I see it, this is one of the unintentional side effects of "hosting in the cloud". If you had co-located servers you'd whack up a firewall and only allow your internal IPs to access non-HTTP ports. Alas everyone now just spins up an S3 image and palms it off to Amazon.
Are you able to make requests between instances on non-public ports? As someone else pointed out Memcached infrastructure typically won't sit on your local webserver.
So lets say you've got memcached, mysql and a bunch of webservers.
On the memcached security group you open 11211 to the webservers group
On the db security group you open 3306 to webservers
On the webserver group you open 80 and 443 to everyone.
* block everything from outside internal network
* open port 80 on web server
* open port 22 on all boxes, but only allow key-based authentication. Oh, and only allow connections from an ip whitelist.
The restrictions on port 22 are probably a little overkill, I admit.
Exposing memcached (or riak or whatever) is operational malpractice.
For efficiency, though, memcached explicitly decided not to handle that kind of filtering itself, e.g. by having the memcached config file take a list of IPs of other memcached servers to talk to (defaulting to none), or including some kind of auth mechanism, and instead relies on the system firewall for that. That's a reasonable design decision, but increasingly unusual. Most servers these days see the system firewall as a backup level of defense, not the primary one, and aim to be secure even when unfirewalled.
If a memcached was bound to non-public interface then it wouldn't be reachable from the internet (and you'd have to explicitly configure it as such). However, if you're running the cache and the app on separate machines, the cache will need to be reachable. For poorly thought out deployments this means publicly reachable.