Have problems with PostgreSQL? Try using Hive (2015) | Hacker News Reader

Have problems with PostgreSQL? Try using Hive (2015)

peter.eisentraut.org

9 pointsaargh_aargh9 comments

This makes me nostalgic of time we were on hadoop stack back in the day.

Eco-system was thriving, hadoop, pig, hbase, hive, mahout. Everyday there was a new tool with an animal name released. Consultancies and VCs were bullish on the data tools. Even cloudera had a tagline that said - "Data is the new bacon". There were "BigData" experts in every organisation. Hortonworks was relatively new and was becoming popular. Somehow, all of them just disappeared in a span of 5 years. The market just shrunk or rather disappeared. Few survived - confluence, databricks and datastax.

Today, I see the same trend in kubernetes space. The eco-system is growing even faster than hadoop's eco-system. I have a feeling that at one point there will be realization that not everyone will need kubernetes and the market will start contracting and consolidating.

Hi, working for major telco here.

I think the realisation is that most data workloads aren't really big data, which people already knew back in 2015, but the same people will overengineer things just to learn stuff. The fad moved onto something else, I agree.

But my department is a giant, just-ETL, everything-requires-a-cluster kind of department. More than half of our business runs on Hadoop, for a reason.

The Hadoop ecosystem is a bit like K8s. If it feels too complicated for your purpose, it probably is. But with all its warts, when your data is big enough, either you use the Hadoop ecosystem or you reinvent significant parts of the wheel.

What tools and stack do you use in your organization?

From major technologies around Hadoop we use Hive, Oozie, Spark... it's easier to say the ones we don't use: HBase.

> So I had this PostgreSQL database that was getting a bit too big

At what scale and for what kinds of applications and use cases does this really happen?

Back in the day, clustering postgres was pain. The focus is mostly on the batch processes here.

Use case primarily is ETL processes and data warehousing

The only problem I have with PostgreSQL is the lack of really good client software, now that pgAdmin is getting less usable with each new update.

DBeaver and HeidiSQL are acceptable replacements, but they’re not specifically designed with Postgres in mind. I feel like the pgAdmin3 desktop application was the peak version of the tool, and it’s been on the decline ever since.

I also feel the same. Unfortunately there's no way back for pgAdmin. But also no way forward to look for. I am convinced the Web based version won't be able to match pgAdmin3 usability ever.

I just forget about it and use DBeaver instead.

IDEA and company have pretty decent DB tools.

Reply on news.ycombinator.com