Comment by peterwwillis | Hacker News Reader

Comment by peterwwillis | Hacker News Reader

Parent Full thread

If you're running out of memory, one would think adding more swap would be the best way to offload the overhead. Hacks aside, if you run into a similar problem again, letting the VM take over will save you time to keep replicating until your load settles down.

Also, if you're using ssh and you need maximum performance you should probably be using an arcfour cipher (`ssh -c arcfour`, alternatives are arcfour128 and arcfour256). I often see a 10x increase in bandwidth using these potentially less-secure algorithms, and slightly lower latency. If you need to save even more bandwidth, a UDP-based SSL VPN may improve things (depending on distance; long links are of course notoriously horrible) or SCTP patched into SSH as a compromise (I haven't played with this yet).

If you're not CPU bound, lzma might gain you bigger savings with marginally higher CPU if you need it down the road.

(I'd also like to take this moment to point out to the Redis devs that 'optimization' of the protocol could have prevented this hack from being necessary until much later. And to the admins of the systems, that monitoring metrics of network bandwidth could have shown the immensity of the traffic for replication. But hindsight is 20/20; let's just remember these examples!)

Surely the fastest encryption scheme would be none at all? It's times like this that it would be nice if ssh still supported the equivalent of "-c none" (or was it "-e none" in v1 ssh?).

Of course, this is based on the assumption that the OP was happy to run their redis sync over the Internet without any other form of encryption. Either the underlying protocol provides encryption or they aren't concerned about their traffic being snooped or MITM'd (albeit very unlikely between two large providers) and so adding extra encryption on top is just an unnecessary CPU overhead.

I've run plenty of (production) stuff over ssh tunnels but it was always nervy with it and happy when the deployment was revamped to avoid the use of ssh tunnels completely.

I'd much rather each underlying application was able to provide tunable encryption and compression itself (using existing trusted libraries such as openssl) as part of its protocol. ssh tunnels are a kludge not a viable production solution (IMHO).

Yes, for a real production solution you'd want some kind of hardened VPN, though I recommended UDP or SCTP based transport for less overhead. But hacks are hacks, and since it's just replication, ssh won't break anything that isn't already broken when it fails.

I would never recommend anyone tunnel anything unencrypted over the internet, especially a database. Arcfour is incredibly fast, comparable to 'none' encryption. RFC4345 specifies the main known attack on arcfour is password auth, so using keys may keep it relatively safe (and using arcfour256 for the longest key).

It's best to assume developers will not implement encryption correctly no matter what library they link to. Always consider how highly vetted the application has been first, and how long it's been around without major issues.

HPN-SSH patches -c none back into the server and client, among other patches that help if you have a long and fat pipe. [0]

[0]: http://www.psc.edu/index.php/hpn-ssh

Adding more swap may not help depending in the workload. If you're using Resque then you need to process jobs faster than they are queued. In that case you also need to ensure jobs are queued on a new instance while the old overloaded one drains (very slowly).

Reply on news.ycombinator.com