1) I'm also curious about your crawling framework, if you care to share any details. 2) How do you compute trending? It seems to show a mix of accounts getting their first backer with those that are already massive. Maybe this idea would help? http://www.evanmiller.org/rank-hotness-with-newtons-law-of-c...
1) Many of the projects I do usually have some sort of crawling aspect to it. So therefore I found myself duplicating crawling code every time and so I decided to turn it into a hosted service that all my projects could benefit from. It stores responses, handles proxyification, and has all sorts of optimizations implemented that's useful and essential to crawling. It passes data via evented webhooks and is built in Rails, PostgreSQL, and Redis. As of now, it's not open to the public but if there's interest, that could change :) For more information, please see:
2) Right now the "trending score" is calculating based on the current and previous Snapshot. The formula is as follows:
current_snapshot_earnings - previous_snapshot_earnings ------------------------------------------------------------------ previous_snapshot_earnings
Very simplified I know :) Your formula is much better and I plan on implementing that soon! Thanks for checking out PledgeSociety as well as the lovely feedback.
what is proxyification?