This is a job for blaze, out of core pandas and numpy frontend to large datasets. http://blaze.pydata.org/en/latest/