The last time I was trying to use pandas, it was the hackernews data dump. It wasn't big. However when pandas started using the memory, my 32GB was just too little.
I just ended to convert the data within postgres, much faster, with sensible memory usage.
I recently moved some data processing from Python/pandas into a database, and with SQL, the processing time went from several minutes to a couple seconds, (and that on a tiny VM).
I understand that not everyone is familiar with databases and SQL, and so default to the toolset they know. But, the performance gains can make learning databases and SQL highly worthwhile. (And, much can be learned in a just few days, especially for those already familiar with working with data.)
From my point of view, the SQL database can store the huge dataset, with its changes, and I can iterate through the results, and make lots of nice queries getting only the data I need.
And not all operations can be done incrementally by iterating through the results.
It doesn't. There's a binary version of the protocol. The output conversion for that is near trivial (transformation to big endian).
https://stackoverflow.com/questions/25962114/how-to-read-a-6...
I have loaded that csv file to postgres, the database had similar size. With indexing it was 15GB on disk. All queries were quite fast.
So instead of loading the data to pandas, and making searches, I just wrote some SQL, and got the same results. However much faster, and with much smaller memory usage.