Because CEO jobs (as with most other valuable things in life) aren't doled out solely based on merit or track record. Mayer is one of the few women in a high-profile position in the Valley, and hiring a female CEO can be good for optics.
Many CEOs have negligible impact.
Judging her on her track record would just be sexist.