BPA is, indeed, a building block for plastics. The media loves to misquote it as a plasticizer, but its not, it's one of the ingredients that produces the plastic. The BPA found in plastics that use it (not all do, or can) is actually unreacted BPA, BPA that did not polymerize into plastic.
BPA is a xenoestrogen, not the only one to be found in plastic manufacturing, not the worst one (many BPA alternatives have higher estrogenic activity), and is also not the only type of chemical to look out for; phthalates, for example, fuck with testosterone metabolism, and are often found in plastics, too.
So, yeah, there is definitive proof of it being a problem, and studies have been published in depth for quite some time. You kinda missed the boat on this by about a decade.