I used to think ruby/rails was the best way to solve most problems, I still do, but I used to too.
80 karma · joined September 11, 2008
I disagree on this point. Starting with a single threaded model allowed my team to scale quickly and with little additional overhead. What we have lost with performance we gained in simplicity and developer productivity. That being said tuning and porting portions of the app to a multi-threaded system is slotted to take place within the next year.
Start with single threaded and simple, move to multi-threaded scrapers when the juice is worth the squeeze.
"This release is dedicated to the memory of our best comrade, Jim Weirich. Thank you, Jim. Rest in peace."