There's this notion from the Supreme Court case Burdick v. United States (1915) that accepting a pardon is an implicit admission of guilt. Therefore, a person can refuse a pardon. There isn't anything in the decision to justify the argument that accepting the pardon is an admission of guilt, it's just stated as a fact. It seems at odds with situations where pardons have been used to correct miscarriages of justice in cases of factual innocence.