That's true, although that requires storage on the other end. Not too much of a problem in most cases, perhaps.
But with single input, single output, you can just do:
ssh destination "ffmpeg -switches" <infile >outfile