HDFS has never been a good idea or implementation, it was just first and rode the fad wave to wide deployment (also see: mysql, php, javascript, ...).

If it were stable and scalable to begin with, there wouldn't be 5 different billion dollar companies offering professional services and customized "fixed hadoop" distributions against the openawfulsource versions.

(from ~3 years ago too: https://news.ycombinator.com/item?id=4298580)