You need to take into account that your interaction with a book is usually short while the effects of your interactions with people accumulate over time.

It might be different for people with very strong personality but I think it is generally true that we are strongly affected by the people we surround ourselves with.