Aside from the obvious analog that your neighbors wifi is literally a light you cannot see, if they aren't maxing out their bandwidth, then it makes similarly no difference that you use it.
It's not 10 years ago, the neighborhood using your router for Facebook doesn't dent the bandwidth.
So how is it not comparable to stray light?