PayFail: PayPal And Its APIs Go Down | Hacker News Reader

PayFail: PayPal And Its APIs Go Down

techcrunch.com

22 pointsvaksel10 comments

Open articleView on HN

Yup, it took me over 30 minutes to get it to work with an Ebay item I just bought.

I was worried it had been hacked.

Funnily enough, I checked their RSS feed for what's going on, and it told nothing.

Then I checked twitter.com/paypal and that was garbage too.

So much for the 'real time web'.

I think they're probably more concerned with bringing it back up at the moment than they are with informing people about what is going on. (I understand that there are more than just engineers working there, but I can understand a PHB screaming at somebody to stop jacking around on twitter during a crisis).

Either that or they hoped that they could rectify the problem before anybody noticed.

This is where your PR department needs to kick in. It's not just engineers that need to go mad right now, PR folk need to join in on the fun.

Starting just before 10 AM it became impossible to authenticate. From about 10:40 to 11:40 they went as far as turning paypal.com into a static page with just this on it:

  We're sprucing things up right now, but we should be done by 11:20 AM PDT.

  Want the details? Checkout the PayPal Announcement Board.
  http://announcements.paypal.com/us/

Sprucing things up? When I finally got back in I noticed some new UI with more gradients+shit. Seems like this is a dev iteration turned cascading failure...

I got the sprucing message too, but it wasn't clear by the time it said (7.20).

This isn't news to me. I've had numerous times where I couldn't log into Paypal, but instead would get Paypal 'This is an error, try again later' messages for a couple of hours.

I also had an item a couple of years ago that I could not pay for. Paypal would just die in the middle of the transaction with a 500 error. I tried for at least a week to get it to work, and Paypal customer service wasn't any help.

This must be some huge corner case failure because I can't imagine that Paypal doesn't have some serious failover/high-availability setup.

Or some interesting human error of course :-)

Considering the massive authorize.net outage that occured because of a fire in 1 datacenter not long ago… well, I no longer have faith that large companies have geographic redundancy ;)

Edit: or, if they have it, I don’t trust that it is implemented in a way that allows an immediate & transparent (or near-transparent) failover.

Amazon does regular "game day" drills that involve things like unplugging an entire datacenter to simulate an electrical failure. It's the only way to really figure out how things will break.

I never have had Chase Paymentech or Bank of America's system go down; but PayPal has gone down many times.

Of course since I'm a pro, I have a queue setup for processing transactions and a backup processor, so the users experienced no problems.

Reply on news.ycombinator.com