Apple has researched this:
> The probability that a random person in the population could look at your iPhone X and unlock it using Face ID is approximately 1 in 1,000,000 (versus 1 in 50,000 for Touch ID). […] The probability of a false match is different for twins and siblings that look like you as well as among children under the age of 13, because their distinct facial features may not have fully developed.
https://9to5mac.com/2017/09/27/face-id-iphone-x-white-paper/
Family members unlocking phones documented previously:
https://www.theverge.com/2017/11/14/16650394/10-year-old-unl...
http://mashable.com/2017/10/31/putting-iphone-x-face-id-to-t...
Face ID updates you face as it changes[0]. So, I guess whenever my brother used my phone it added my brother's face features since algorithm probably found some similarities in face features.
[0] https://www.theverge.com/2017/9/27/16373710/face-id-iphone-x...
If they are sufficiently similar the phone could theoretically expand its knowledge of what A looks like so that B’s face triggers a positive match.
I have no idea how close in appearance they’d need to be for Faceid to do this though.