Yes, using your connection will always cause an impact. If it's noticeable or not depends on a bunch of factors that are hard to guess about. I wouldn't run it very often if I didn't have some kind of QoS in place.

If you have a good router that does some queue management and doesn’t suffer from buffer bloat, it shouldn’t impact it too much.