It will affect the latency on the actual internet link. The second you add extra traffic latency shoots up...it doesn't matter where that traffic comes from. You can mitigate the effect somewhat with QoS but it doesn't go away.

Bufferbloat on the upstream pipe, if somebody is using your shared connection.