Example: you're not keeping the server up to date, and someone injects Javascript into your payment page. You could be liable.
As I understand it (admittedly not well), PCI is kind of like HIPPA/OSHA, in that there's some explicit no-no's (e.g. not using SSL, leaving patient info lying around, or not wearing hardhats) but the more important point is about maintaining a "culture of compliance".
http://usa.visa.com/merchants/risk_management/cisp_merchants...
Requirements scale with processing volume, and are generally minimal for merchants processing under 20k Visa transactions annually.
Many gateways use tokenization to dramatically reduce PCI scope for their merchants. It's fairly standard, actually. Even with tokenization, merchants have compliance obligations. The required network scans, for example, protect consumers from merchant websites being compromised ahead of the tokenization step.