What surprised me (last night as I wrote my paypal integration) is that Paypal's API is broken by design. Horribly.
Even their new APIs are based on the IPN "reliable messaging" notification system. But there's nothing reliable about it:
* You receive a message from Paypal
* You post it back to Paypal
* You get back "VERIFIED" or "INVALID"
* The burning stupid: This postback is also what acknowledges the message and stops the retries.
In other words, your code must either:
1) Commit transactions in your database based on unverified information, then figure out how to revert the changes if the message comes back invalid. Or:
2) Verify the message (stopping retries) and then try to commit your transaction, possibly failing and losing the IPN message entirely. Your customer may never get his product enabled.
Oh, and let me mention that the verification postback can happen only once. It's impossible to build an idempotent handler for IPN messages.
Paypal's documentation and sample code push you towards solution #2. This is just negligent. I've built porn-serving infrastructure that was a thousand times more robust. People trust this with real money?
And don't get me started on Paypal's near-useless morass of documentation. And the fact that IPN messages don't have any kind of unique identifier.
Technology FAIL.