If the probability that a person is "in on it" is P, then the probability that N people are "in on it" is P^N, which converges to 0 as N goes to infinity. For every library you use, all of the independent people providing feedback, reviews, stars, etc. would also have to be "in on the" conspiracy.

You realize that the open-source code you use has hundreds of stars on GitHub, tons of independent people providing edge case and bug feedback. For all of those people to be bad actors is unlikely. If something was malicious about it, that information would spread through the web like wildfire, and you would know about it.

This is why people are reluctant to use libraries which are not well-known or have very little usage feedback. If you do use these libraries, you will probably read the source code first.