There was a WHO study published just last month[0] which did not find any dangers associated with microplastics in drinking water. One of the conclusions mirrors your reasoning:

"Although it is not possible to draw any firm conclusion on toxicity related to the physical hazard of plastic particles, particularly the nano size particles through drinking-water exposure, no reliable information suggests it is a concern. Humans have ingested microplastics and other particles in the environment for decades with no related indication of adverse health effects."

[0] https://apps.who.int/iris/bitstream/handle/10665/326499/9789...