What I'm wondering is how many blockchains are you verifying as a peer. When you download the blockchain software, you start verifying the entire blockchain, do you keep getting blockchain hashes from other peers and keep comparing them to see what's the consensus?
You verify that they're all valid, and the blockchain with the greatest accumulated difficulty (correlated with chain length) is assumed to be the valid one used by the whole network.
Whenever you get new blocks you first verify validity, and then you check if they make up for a conflicting blockchain longer than yours (if so, you switch), shorter than yours (ignore), or if it extends the blockchain you have (then you add them).
Number of peers don't matter in Bitcoin.
This typically also means that the longest individual blockchain wins.