Part of the performance record might well be due to not everyone taking the pill remembering to always take it. If you miss the odd day, I'd imagine that the chance of accidental pregnancy goes up a fair bit.
Also having to take it the same time every day, and the interaction between the pill and antibiotics. I've heard that many women don't realize that most antibiotics destroy the effectiveness of the pill, meaning that a back-up birth control method needs to be used in a cycle where she gets strep, or an ear infection, or a UTI, or many other common illnesses.
Yes, it was always my understanding that the 99%+ efficacy of oral contraceptives were based heavily on taking them in the manner prescribed.