Just to add to this, it sounds like OP didn't contact Affirm until after they already started the charge back. I definitely don't think it is appropriate to do a charge back before contacting the vendor that got the payment, which is Affirm.