That last part is the relevant part: either you use FBA or you loose visibility/sales. That combined with the dominant position of Amazon seems, from a layman perspective, a textbook case of abuse of market dominance.
I would have to look back through my order history, legit I thought prime was always fulfilled by amazon, and that was how they could guarantee the shipping process. But amazon's page even talks about why FBA might not be right for your products (too bulky large etc., i.e. whatever isn't efficient for amazon to store/distribute). But then amazon probably was not content to say "that entire class of products cannot be Prime".
So my view to this was, amazon is selling a product (FBA) and part of what you get is use of the Prime label. But, certainly willing to cede being mistaken if that is not the case.