Scaling to 2M API request per day

stackshare.io

6 pointsmttsn3 comments

Just finished the article. Great read! How are you guys able to retrieve data instantaneously for the personal API? I understand that you can cache the company API but how are you guys caching the Person API as there will be billions of users online across all social networks.

Quoting Harlow here:

Great question! When we don't have an email address (or company domain) in cache we use our robust worker platform to doing live lookups. We leverage Sidekiq Pro to process millions of Lookup Jobs per day. We can typically finish a Person Lookup in under 3 seconds by leveraging the massive concurrency that Sidekiq affords us.

Great answer! Thank you for your response. I am a fanboy of Sidekiq and I enjoy its speed and ease of use. What are some places you guys look for personal info. Are they just APIs or do you do some social network scraping? Thanks.

Reply on news.ycombinator.com