To make sure there isn't confusion here, having card details go through your server, even if you aren't saving them, still can lead to certain PCI compliance burdens (e.g. you may need to get an audit from a PCI auditor verifying this).
Having the card never go to your server is the best way to make sure you are PCI compliant, as you mention.