When pandas can’t manage a data set, I would convert to hdf5 and use the vaex pandas-like API. This is very fast and memory efficient (out of core) for my typical 40-70GB data set size and reportedly remains fast up to 1 TB.

The key ingredient is the on-disk format: hdf5 and more recently also arrow.

Looks like DuckDb would force me to use SQL, which is not what I typically want. That being said, if this offered better data management than a binary blob like hdf5, it might be worthwhile. If the use case requires processing historical data for example, a real DB would be better than juggling 10 x 40GB files or having one big 400GB file that keeps growing.