Are you/your dad Japanese? I feel like we're missing most of the context here.

Skin on fruit is full of fiber, along with potatoes, etc. I only ever take the skin off things if it really bothers me.

My mum peels button mushrooms, never realized it was weird and no one does this until I was an adult.