Yeah it removes all of the random "liked by someone you follow" or "popular in your network" etc tweets that slowly creep in over time.
Yesterday Twitter started showing me "someone you don't follow replied to another person you don't follow" types of conversations and it's gotten really absurd at this point.
This happened to me yesterday as well, only it was "liked" instead of "replied". In my confusion I clicked on the account, since I didn't recognize it. Only to mistakenly click the Follow button, as my brain didn't manage to correctly derive the current status of it. Because I actually thought I had accidentally followed them or something similar.