Twitter experiences major, 2-hour+ outage

This is a perfect storm. Many twitter clients automatically retry posting tweets when there is a non 200 return code from the Twitter API (including humans posting on the website seeing errors). Unfortunately, Twitter has been returning 5xx codes while successfully posting tweets. This, plus the apparent fact that duplicate tweet detection is failing/turned off means that all of the duplicate posts get through. Mayhem ensues...

This happened the other day as well (probably on a shorter timescale, I'm not sure). I'm not sure if it was a 5xx error code - it seemed more like connections were hanging.

It showed me the danger of working on someone else's platform, cue warning story:

I had something that queried Twitter in php (Curl) but it never really occurred to me to time out the connection (this was ages ago, I forgot about the project till the other day). For some reason I assumed the host would kill long running PHP scripts..

Apparently not because my whole account got disabled for too many open connections.... oops.

"..failed enhancement of a new approach to timeline caching"

I hope this is not a testament to scalability of Cassandra or Redis.

Ref: http://engineering.twitter.com/2010/02/link-cassandra-at-twi... http://github.com/twitter/haplocheirus

I noticed I kept getting Internal Server Error messages, so I kept trying to post, until I checked my timeline and saw it showed my post 4 times. Then I saw someone I follow post the same message about 20 times in a row.

It kind of reminds me of the Win32 exception with a result code of 0, "Operation completed successfully." but backwards.

Never before has a free service been relied upon by so many diverse people, from business people, to students and demonstrators. Long live Twitter!