"alone did not improve the performance much, but by only taking every Nth sample, I could lower the effective sample rate, shorten the processing time, and reduce the number of bytes I had to send back which greatly improved the latency."

Right, the latency would improve but with the cost of increased uncontrolled noise. An anti-aliasing filter shall be applied before the decimation to overcome that.

IIR filters are out of the question for this purpose because of how the phase is affected.

FIR filters tend to be costy due to a possibly high order. Proper decimating filters reduce that cost by a factor of N. Also if it's not required to keep full Fs/2N bandwidth clean just a part of it, it's enough to just "punch holes" instead of doing a full lowpass, which reduces the cost further.