If you change your walking habits based on what you think others think of them, it might be a good time to reevaluate how important external validation is for you. I'm not saying this to be condescending, it is my personal and very subjective opinion that it doesn't seem like a healthy approach to living a life. There's nothing wrong with matching the pace when you walk with someone else, your daily habits are something else, though. You should decide what's good for you, not the opinions of others.