In my opion, Cephfs and GlusterFS have much better support for Pandas and others, you don't need change any code between local development and train an distruted way, rather than a path.

Second, the Python library for HDFS provides only subset of filesystem API, which limits what you can do with it.