Amazon AWS Throttling Non-Elasticache Memcache Traffic

We've been trying to set up memcache/couchbase boxes on Joyent East and Serverbeach Virginia today and noticed that we get a max of 300 reqs/sec per virtual instance when we send memcache requests from Instances on AWS East.

It doesn't matter if we change the security group on AWS or if we change the port of the memcache traffic. the traffic itself seems to be inspected and throttled.

This is a MAJOR problem. Basically all AWS customers will need to start testing to see if their specific type of traffic is being throttled.

Within the AWS cloud we see 3000+reqs/sec between boxes. But once they leave amazon it gets really slow. From Amazon West to Joyent East and Serverbeach Virginia we saw a further drop to a total of between 6-12 reqs/sec using the memcache protocol.

Sending data over ssh tunnels still works fine. A scp file transfer from AWS East to Joyent East transferred at over 22MB/sec.

To replicate this:

Set up a box outside of AWS running memcache. Default settings are fine. And on Amazon set up a box with php installed and the memcache pecl extension.

And then run this php code. replace 111.11.11.1111 with the IP of your memcache box:

<?php

$memcache = new Memcache; $memcache->addServer("111.11.11.1111", 11211); $time_start = microtime(true); for($i = 0; $i <= 1000000; $i++) { $memcache->get("d.{$i}aaaaaaa"); if ($i % 1000 == 0) { $time = microtime(true) - $time_start; $items = $i;

        $rate = $items / $time;
        
        echo (number_format($rate, 2))."\n";
    }

}

?>