It's concurrent in that multiple crawlers run at the same time. But I do not concurrently crawl single websites. An initial version had that, and Magnetissimo easily crawled 600 torrents every page 3 seconds (not hyperbole). It just pounded websites and I didn't want that. It was bad karma.