Paypal has not been sending payment notifications to merchants | Hacker News Reader

Paypal has not been sending payment notifications to merchants

discuss.tindie.com

132 pointsemilepetrone71 comments

You can make an API call to check the status of a transaction very easily.

When my users are redirected back to my site (thanks page, or similar), I check if their transaction is completed, if not, I kick off an every-five-seconds check while asking the user to hold on while we talk with paypal. I will eventually fail after some number of checks of course, but this means PayPal can stop sending IPNs and everything will just keep going along just fine.

If the user might not end up back on your site for some reason, run a cronjob that tries to verify transactions created in the past day/hour/whatever.

An issue like this doesn't have to, and really shouldn't, cripple your business.

Agreed. It doesn't take a lot of working with PayPal to know the IPN system is fragile at best.

PaymentDetails is mandatory for any adaptive payments integration.

https://developer.paypal.com/docs/classic/api/adaptive-payme...

Agreed - the statement of "Because of how critical IPNs are to any Paypal integration, we didn't think to double check if IPNs were not being sent." is exactly why you should double check - anything that is critical needs tests and backup solutions.

Belt and suspenders, dude, belt and suspenders.

When dealing with PayPal, go for an elastic waistband as well :)

That's very close to what I've always done. I have a notification to warn me if I have new failed sign-ups due to payment and keep Paypal responses to catch all sorts of problems. It's just good practice to expect things to fail sometimes. That's also why I have my status page hosted with a different provider and twitter status as a back-up for that.

My favorite Paypal annoyance is that for split payments there's no way to verify that a seller's Paypal account qualifies for split payments (has to be a business account and verified), without trying to send a payment. At least, that's the way it was last it mattered to me.

Actually you can check if an account is verified!

Here's a snippet in Ruby I use to check. It's not perfect, but it catches a lot of cases:

    # ...

    def initialize
      @request = PaypalAdaptive::Request.new
    end
    
    # ...

    def verified_status( opts={} )
      begin
        opts.assert_keys! :first_name, :last_name, :paypal_account_email
        result = @request.get_verified_status(
          emailAddress: opts[:paypal_account_email],
          firstName: opts[:first_name],
          lastName: opts[:last_name],
          matchCriteria: 'NAME'
        )
        Rails.logger.debug "PAYPAL VERIFICATION: #{result.inspect}"
        return result.success? && result['accountStatus'] == 'VERIFIED'
      rescue
        return false
      end
    end

Here's the API docs: https://developer.paypal.com/docs/classic/api/adaptive-accou...

What I can't stand is that accounts in some countries (at least India) can't send payments from balance. So you can't get your fee. This is mentioned nowhere in the docs, a customer just hit it one day and that's how I found out. I asked paypal what other countries can't send payments and they said... wait for it... they "don't have a list". I kid you not: https://twitter.com/rfunduk/status/412980259001102336

If you've ever sat down and looked at Paypal's APIs properly, for instance with a view towards a proper risk analysis and/or from-scratch integration, then this should not surprise you. I did this for the first time perhaps ten years ago, and here's what I found.

Firstly, the degree of marketing bullshit that you have to push through just to get to the truth of the APIs is staggering.

Secondly, they perform backflips to avoid telling the truth about their integration options, which is to say that they're all fundamentally insecure for real time digital goods and services unless you implement IPN plus polling plus duplicate detection / additional round-trip validation calls, ie. you can't trust an IPN notification. The degree of complexity (order state tracking, IPN state tracking and duplicate detection, retry support, running your own IPN-receiving server) and latency requested here of client businesses is immense.

Thirdly, their idea of international support is pathetic. It seems that they've basically duplicated their entire business process to other countries, translated it, and then assumed that everyone in that country requires only one human language in all of their interactions: documentation, support, emails, etc.

Finally, as is widely known throughout the industry they have a shocking reputation for the arbitrary suspension and seizure of accounts and assets, with little to no recourse for those affected.

I am not surprised that Stripe has taken off. Unfortunately, that's still lipstick on a pig: fundamentally, the settlement, risk, transparency and government interaction model of credit card networks means they are unsuitable for an increasingly large volume of business around the world.

Agreed. Over the past 9 years I've had to implement double, triple and end-of-the-month checking on PayPal to be sure that we have some reconciled idea of reality with respect to payments.

What's worse, their system essentially rewrites history over time. This means that rather than keeping a ledger of payment events, they change the dates and amounts depending of payment status / reversal /etc. Querying later in the week and the data is different. For the same charge.

I ended up keeping an event log of everything we ever get from them (IPN, API polling, reports) and synthesizing that into a current snapshot at query time. The code is riddled with comments saying, essentially, "this charge, then return, then recharge and failure means that the payment is OK".

I would switch to Stripe if they offered micro-payment pricing like PayPal's 5c + 5%.

https://news.ycombinator.com/item?id=7871180

Notice the questions I asked.

Not sure if this is related to this particular case. From previous experience, PayPal will silently stop delivering IPNs if your webhook URL ever returns a non-200 status code for an extended period of time.

It took us a fair amount of time to figure out that this was the cause of our issue -- the PayPal UI didn't indicate any problems and PayPal business customer support was unable to see the source of the problem either.

Interesting - we have still gotten IPNs which is why we didn't suspect anything.

Have you checked your web server logs? We had a problem a year ago where we were getting 403 errors (forbidden). It turns out that paypal's DNS was a bit screwy, so apache wasn't able to do a reverse dns lookup on the ip address, and it was therefore getting rejected by our server. I simply added the ip addresses of all of paypal's IPN servers to our Allow list, and we haven't missed any IPNs in the last year:

Allow from .paypal.com

Allow from 173.0.81.1

Allow from 173.0.81.33

Allow from 66.211.170.66

What about if it finds 301 or 302 codes (redirects)? I hope they don't count as a negative...

I wouldn't chance it; the first time I implemented IPNs, I had to spend almost a week on R&D aimed purely at discovering and working around all the undocumented fuckups.

Webhooks are not hard. That Paypal fails at them this badly makes me think their infrastructure is a (much larger) disaster waiting to happen; if I still did freelance work, I would be moving all of my clients to Stripe right now, before the next inevitable mishap.

They actually do the right thing with a 301 redirect, and re-post to the new URL. I did a site wide http->https redirect, and surprisingly the IPN script continued to work.

Yea; we always, always return a 200. I have no idea about redirects but you can test that using the (very, very painful) sandbox. FYI, another lesson we learned the hard way: if you are testing your IPNs in the sandbox and you include a port number in the URL, the IPNs will silently fail. You also can't use IP addresses for IPN testing (again, they will silently fail) so I setup an A record in our DNS pointing to my local machine for testing.

IPNs are annoyingly difficult to use.

Some accounts simply will not send them until an IPN URL is set, even for transactions that include the notify_url in the request. You cannot even see the IPN history until you enter an IPN URL, even if it was used through the notify_url param.

IPNs that include UTF8 characters will not validate by default in most PayPal accounts. PayPal defaults to Win1252, translates the UTF8 content to Win1252, tells you it's Win1252 and then seems to validate the IPN against the original UTF8 data. I have never been able to validate a Win1252 IPN that had UTF8 content. I've tried iconv, stripping bits, and plain substitutions (ü -> u). Nothing seems to work. This seems to be getting better though.

For about 2 months last year they forgot to include a field that had to be added in order to verify the IPN. And you had to iterate through a few different values to see which one was correct. So we got a rash of support tickets asking why our customers were suddenly getting notifications of issues verifying PayPal purchases and were having to manually approve orders. I submitted a ticket after a day of dealing with the issue and got a response about 2 weeks after they fixed the issue. There was no indication of the problem on their status site.

The IPN verification service randomly returns INVALID for valid data or simply doesn't work. Sometimes IPNs are delayed.

Dealing with the PayPal APIs are far, far harder than their sales sites would have you believe.

I banged my head pretty hard on this recently, but afaict I have a working solution. I wish this had been better documented. One problem I faced is that I had to fiddle with Flask to make it work (Python user here). I'll post my setup somewhere tomorrow and comment back here.

Here is my solution to the encoding issues, let me know if it works for you (assuming your environment let you access the body of the request without parsing it, or already knows how to parse it correctly).

http://blog.cyberion.net/2014/08/parsing-paypal-ipn-requests...

Thanks, but that's a wholly separate issue from what I was talking about.

When you have UTF8 data in the IPN, PayPal translates it to win1252 in the IPN. But it verifies the IPN against the original UTF8 data. The only way to fix it is to go into the profile and change the IPN format to use UTF8. There is no programmatic way to set the encoding for IPNs to be sent in and there are no workarounds that I am aware of.

If you never see international names or addresses with accents and diacritics you won't run into the problem.

That's not my experience, validating the data has always worked by just spitting it back out to them without any alterations to the data.

Here's an idea, stop using shitty services like PayPal. Today it's IPNs failing, tomorrow they're locking your account and holding your money for 180+ days with no justification.

As much as people complain about them, I still haven't found a service that matches their reach or ease. Let's face it, PP is a de-facto bank without any of the regulations, requirements or safeguards.

ease of use is still kind of key for me. I pay a lot of people in Europe / Asia and if it weren't for paypal I'd literally have to make a trip to my supermarket Western Union twice a week.

Yes, exactly. When I tell people where they need to send funds, their first question is "do you have a PayPal account?" Western Union, Moneygram and such are all highly inconvenient for both parties.

And services like Stripe aren't an option for me due to their restrictions on use as well https://stripe.com/us/prohibited_businesses

My work covers multiple items on that list. Compare that to PayPal's https://www.paypal.com/webapps/mpp/ua/useragreement-full#9

Edit: I should mention, of Strip's list of "prohibited businesses", nothing I do is illegal or unethical. Just in case I gave that impression.

wow that list is huge!

Hell, I had no idea that Stripe's prohibited businesses list was so ... massive.

The list runs the gamut from "risky" to "value judgement", and I can't help but wonder how all those items ended up on the same list.

Some are clearly legal liabilities. The rest almost certainly have high fraud/chargeback rates. Can you imagine how many people must dispute what they paid to a psychic

What about "48. Telecommunications equipment and telephone sales"?

What about it? Telephone sales are high risk.

Yes. A quick google reveals:

> Telecom – The telecommunications industry is much more proficient with technology than it used to be. However, once services have been provided, recipients may dispute them. Card-not-present transactions are a mainstay for telecom companies. http://www.aliantpayments.com/who-we-service/high-risk-proce...

I would imagine that things like VoIP service are targeted by criminals who constantly need new, anonymous phone numbers for other scams.

Stripe's real business is measuring and assessing risk. The moving dollars around is the easy part.

I was thinking racks of Alcatel-Lucent and stacks of Sagem, but perhaps you have a point.

I work at Stripe and just wanted to give a little bit of color here in case it's useful. These restrictions come from our financial partners which unfortunately means that we generally don't have much flexibility around them.

If you have any questions about any of this or want to see if we can work with your business, please don't hesitate to reach out to us at support@stripe.com.

Wow - "Engaging in any form of licensed or unlicensed aggregation or factoring" Lucky I don't use Stripe to sell my scientific software which does unlicensed factoring of large matrices and also sell "Personal computer technical support" to help customers who have problems using it.

But the real problem with Stripe is simply that it's not available in almost any country. They're too small to have the resources to read the laws and open bank accounts outside America.

I think they meant this kind of factoring: http://en.wikipedia.org/wiki/Factoring_(finance)

I have utterly no idea what several items on Stripe's list are. Some that I tried looking up remain, at best, unclear to me.

Some of the items are extremely vague, even to the point of contradiction, or to the point of sweeping in huge amounts of activity that nobody should consider high-risk. For example:

"3. Virtual currency that can be monetized, re-sold or converted to physical or digital products or services or otherwise exit the virtual world"

So, implicitly, then, virtual currency that can't be easily converted back to real cash is fine. But:

"43. Quasi-cash or stored value"

This could describe a virtual currency that doesn't violate #3. But then:

"52. Selling video game or virtual world credits (unless you are the operator of the video game or virtual world)"

Make up your mind, Stripe.

Meanwhile, #2, "Weapons and munitions". Under US law, a PlayStation can be treated as a munition.

#4, "Sexually-oriented or pornographic products or services". So, no sex-ed materials, bad romance novels, or condoms.

#15, "Age restricted products or services", can be read to preclude any business that restricts signup by under-13s in order to be sure they meet the requirements of US federal law.

#31, "Extended warranties". So, this precludes pretty much everybody who offers a product customers might want long-term support for. Like, say, every computer manufacturer in history.

#41, "Prepaid phone cards, phone services or cell phones". This means a hypothetical electronics store can't sell cell phones -- even simple unlocked phones without plans.

#48, "Telecommunications equipment and telephone sales", would preclude the sale of any sort of networking equipment. Or, really, any piece of equipment capable of connecting with a network, including any general-purpose computer.

Companies should put the same care into the design and usability of their terms and conditions as they do into their products, because the former has a direct impact on the latter, and often makes for huge PR messes.

(I work on risk at Stripe.) Most of that list is bank legalese we're contractually bound to include in our ToS. This is not to say these restrictions don't mean anything; just that I wouldn't interpret them too literally. We've actually just created a much more coherent version of this list, and it will be up on the site as soon as we can get our banking partners onboard.

As for your specific case, please email me at anurag@stripe.com and we can try to figure out if Stripe can somehow work for you.

In Europe (and I think some states) Paypal is a fully licensed as a bank.

You need to understand that there's a large slice of users who use PayPal exclusively, because they perceive it as a safer online payment option. These users see a credit card payment form (say, hooked up to Stripe), they panic and they start sending "who the hell do you think you are to be asking for my credit card number?" emails. We tried arguing, we tried explaining that we don't see the CC info and that we are getting far more personal data on them when they use PayPal, but it was all pointless. They don't care. They want PayPal.

So the question is if you want these people's money or if you are willing to take a hit in sales just to stick it to PayPal.

Depends heavily on your target market.

I thought this, so when I moved to stripe I retained a pay with paypal option (and braintree makes this very easy now) but it was hardly used so I just got rid of it. I still use paypal for some really old stuff that I can't be bothered to update but all my new stuff is stripe only.

We have been experiencing sudden weirdness with out PayPal Pro payment processing integration with our Magento cart over this time period. I have spent hours looking over logs and on support call with PayPal integration support. You would think that since PayPal and Magento are both part of eBays X.commerce that they would work together well.

I received a lot of "hmmm that's weird" and "well i've never seen the API do that" comments from the integration support specialist at PayPal."

What is happening for us is that Paypal is simultaneously sending a authorization and then a second later a capture. the results are weird some transactions process and other get a duplicate invoice it error.

This has always been a problem with Adaptive Payments, just not always so severe. Paypal's Express Checkout product has the best solution to this in that a payment cannot be completed without an explicit API call by your application. This makes the payment confirmation IPN redundant.

It is a more complex integration, but our product has been much more stable since we made the transition to Express Checkout.

This does not however prevent other problems with IPN delivery on refunds and other post order activity. Paypal's IPN infrastructure is pretty weak in general, very poorly documented and completely untestable.

The issue here is that Express Checkout isn't available in all countries. When I was integrating PayPal for the startup I was working at last year we had to use Adaptive Payments because of this. I don't mean some small South American county nobody has heard of, the business was based in Ireland.

As other commenters pointed out we didn't have much choice of providers because of our business model (hotel booking). Stripe and Braintree didn't want anything to do with us, so we went with PayPal to start with. I think they are now using RBS WorldPay, but even that had it's issues...

This is a mission critical bug for any other startups that depend upon the Adaptive Payment API. That may only be a subset of Paypal merchants, but for those that do - you need to know this ASAP.

I don't remember accurately the documentation URL but there was a mention (I did lots of PayPal API development) that IPN cannot be always trusted and you should instead query the PayPal API (in a cron job fashion) to check if the transaction went through.

There are many reasons why an IPN will fail. Like having your firewall settings messed up, network issues on your or PayPal's side, PayPal actually sending an IPN but your infrastructure not making use of its information...

If your business process relies completely on a callback that can fail for plenty of different reasons, then your business process is broken.

Except that part of PayPal's business model is saying that you can depend on that callback. This isn't on the business here, it's on PayPal.

Now, I'd agree if you said that PayPal isn't suitable for a business to use, it should be relegated to the realm of individuals and small businesses who don't know better, but that's different than what you seem to be saying.

Can you link to the part of PayPal's that mention you can depend on IPN?

If your payment processor builds a product offering around a webhook which tells you when someone has paid for something, and that webhook fails hard at their end for over a week while no one there even notices, then your payment processor is broken.

One anecdote is hardly evidence of their service being down for a week, more likely it was an issue with the merchant site or some intermediary. Having experience running a service similar to PayPal's IPN, the most common problems are invalid certificate chains on the webhook URL or some other misconfiguration or blacklisting. Unfortunately webhooks are usually work most of the time that people dont think they need to implement a secondary system to check on pending transactions.

I'd be very surprised if IPN was down for several days, much less 9. Can anyone confirm such a thing?

IPN hasn't gotten much love in years but still pretty cool that it was (one of) the original webhooks back in 2001 and has worked decently for the past 13 years. In fact, an integration from 2001 should continue to work OK.

It's certainly not generally down for everyone. I've not seen any oddities in IPN for a few years, and can confirm notifications have gone through over the past few days.

It's important to be careful about jumping to drastic conclusions when debugging a connection like this.

E.g., from the OP:

> I called Paypal Technical Support to verify exactly what we were seeing. Sure enough, they had received other calls today about this exact issue. Other vendors were not receiving IPNs for their orders. The ticket for this "critical bug" was created today.

> The problem with that fact is that our customer's order was from 9 days ago. For at least 9 days (we are going through all of our partial orders now to see the full extent of this bug), Paypal was not sending IPNs to merchants and did not know.

This doesn't actually follow logically. The tech support rep mentioned getting other calls about IPNs not going through does not mean NO other IPNs were going through. Likewise, if Tindie has actually not gotten any IPNs for 9 days, that doesn't indicate anything about the other millions of PayPal merchants.

This reminds me a bit of customers who call in a panic that "your entire website is broken!", and after a bit of discussion it becomes clear that their internet connection is down.

The same stream of events could be caused by lots of things, including minor configuration changes on Tindie's end.

IPNs were unaffected for us. This probably only affected a small segment of customers.

I organize a bunch of trail running events, our registration page has both a stripe and paypal option; 85% use stripe, but the other 15% I don't want to give up on. Oh, and no IPN glitches here in the last 2 weeks (or ever actually - knock on wood).

a) Stop using paypal (or at least make it an option but recommend stripe or similar)

b) IPN is clearly going to be flakey, there are several reasons why IPNs might not be delivered and they are not all paypal issues. In the paypal docs it says that if your webhook doesn't return 200 to some IPNs then it will stop sending them. For my new stuff I don't use paypal at all (stripe is just too good) but when I did I had a cron job running every 5 minutes that checked every transaction that had been created but no IPN had come through for. Over the cause of a couple of years I caught several transactions where the IPN had been lost.

What is up with that blog title?

> Disbursements through US Bank Accounts or Debit Cards

Is it only related to US bank accounts and debit cards? Or is it an IPN issue affecting all card?

Title is Tindie specific... I wrote it for our audience but the news I thought HNers would want to know

This explains a lot... my IPN handler had been awfully quiet, but thankfully it was only used for automatic thanks.

had this problem for years, along with google checkout's callback process. google was worse. they seemed to get to a point where if/when they'd ping, if the request took longer than a few seconds to reply, it would stop, then never retry.

Google Checkout was awful to work with. It was hard for the vendor to setup and use. It was hard to program against. It was hard for the customer to use. It was hard to test. And of course it had Google's famous lack of support.

Speaking of support, we qualified for the integration reward ($1500 or so; maybe more). We tried to claim the reward for over a year. Every time they'd say "yeah you totally qualify!" and we'd ask how to claim the reward and then silence. Of course every email response took 3-7 days. A few months later we'd go through the same process. Still waiting on that check to arrive.

We supported it for about a year and eventually gave up. Payment notifications could take up to a minute to arrive. You couldn't specify the tax (or shipping from what I remember) for an order in any sane way. For a while it was over 50% of our support tickets while being used by less than 10% of our customers and less than 1% of all transactions.

I was so happy to delete it from our codebase. And we won't support any payment processor they have to offer anytime soon.

For me it was.... either instant or 15 minutes. The 15 minutes was 'security check' stuff, apparently. But I was selling digital downloads. I'd just have to approve them instantly - no one expected they'd have to wait 15 minutes to get approved, even when I tried explaining it. So... I had some bad charges come back, but the download was already done - no way to protect myself. GC was pretty bad.

BTW this post was #7 on HN, and then the HN gods flagged it for banishment. Not sure why...

It set off the ring detector. It looks like a false positive to me, so we'll turn that off.

We also changed the title (from "If you use Paypal, you should know they aren't sending IPNs") to make it more neutral.

Ring detector?

A voting ring is when you submit something, and then tell a few dozen of your "friends" to vote for your submissions (and you agree to do the same for them). HN has heuristics which detect this sort of behavior, and penalizes the ranking of an article that looks like it's being voted up by a voting ring.

(Disclaimer: I haven't read the HN code, and have no idea how they actually detect such things.)

Sorry—a bit of local jargon. Fortunately gknoy has explained it.

Not to worry. Just the NSA deploying a MITM attack. The bugs will be ironed out soon and your transactions will proceed smoothly as before.

Reply on news.ycombinator.com