Have you timed it? Starting threads in Python is slow...
$ python quicktest.py
['http://www.google.com', 'http://news.bbc.co.uk', 'http://news.ycombinator.com', 'http://www.cnn.com', 'http://www.foxnews.com', 'http://www.msnbc.com']
[<Response [200]>, <Response [200]>, <Response [200]>, <Response [200]>, <Response [200]>, <Response [200]>]
Serial: 1.23853206635
[<Response [200]>, <Response [200]>, <Response [200]>, <Response [200]>, <Response [200]>, <Response [200]>]
Multiprocess: 0.912357807159
[<Response [200]>, <Response [200]>, <Response [200]>, <Response [200]>, <Response [200]>, <Response [200]>]
Multithreaded: 0.708998918533
edit: Here's the code: import requests
import time
from multiprocessing import Pool
from multiprocessing import Pool as ThreadPool
session = requests.Session()
urllist = ['http://www.google.com',
'http://news.bbc.co.uk',
'http://news.ycombinator.com',
'http://www.cnn.com',
'http://www.foxnews.com',
'http://www.msnbc.com']
# Warm up?
responses = []
for url in urllist:
responses.append(session.get(url))
print urllist
start = time.time()
responses = []
for url in urllist:
responses.append(session.get(url))
print responses
print "Serial: {}".format(time.time()-start)
start = time.time()
pool = Pool()
responses = pool.map(requests.get, urllist)
print responses
print "Multiprocess: {}".format(time.time()-start)
start = time.time()
pool = ThreadPool()
responses = pool.map(requests.get, urllist)
print responses
print "Multithreaded: {}".format(time.time()-start)This is certainly one of the cases where you should just do whatever is simplest (to _you_ the programmer). The first step is always to optimize for cognitive overhead. I.e. make the code easy to reason about. Next (and relatively rarely) is it necessary to good to optimize for different bottlenecks in your code.