You must have set it up wrong. It doesn't add any latency, but you do need a higher bitrate to get similar quality.
"Lost packets can be replaced with loss concealment by calling the decoder with a null pointer and zero length for the missing packet."