We have written distributed crawler using celery/Rabbitmq.

Analyzing social signals real time and it is processing more than 10 million requests everyday. On the way to scale it to process more than 50-100 million requests per day.

Other than that doing nlp tasks along with machine learning using topic modeling, ner etc.