It's not the office hours themselves that felt unfair, but the way some students got privileged hints about upcoming exams.
put another way, they showed up and asked questions and got info -- and you didn't. that's not privilege, that's effort
To answer your question, I don't recall ever going to "office hours", as I was generally a top student with minimal effort and an autodidact, but I learned about it through friends. Having parents who are both professors also gave me a front-row seat to how common this was.