Perhaps the confusion comes from a particular set of facts that would seem to contradict what you are saying. Say company 1 is first to file for the patent. Company 2 has been working on the same product, but files late relative to company 1, so loses out in getting the patent. At the same time company 2 was, in fact, the first to invent.
Wouldn't the earlier work performed by company 2 be prior art, and be used to invalidate company 1's patent? In that event, what is the value of a first-to-file system, if a competing inventor can use their earlier invention as prior art? And if a competing inventor can't use their evidence of earlier invention as prior art, why not? Perhaps it's simplistic, but common sense would seem to dictate that for "first to file" to mean anything, prior art would have to be disregarded.