Hacker News Reader

Comment by Sevii | Hacker News Reader

Postgres fills a niche that has a lot going for it. It’s standard SQL with transactions so it’s easy to code against compared to Nosql like Cassandra. There are lots of powerful extensions like JSON and full text search. And you can scale it pretty far.

View on HN

Postgres is likely suitable for 90% (or more) of the crap "engineers" currently fall all over themselves to implement Kafka, Spark, and whatever other bizzword laden "big data" or "machine learning" resume driven development for.

It's not a niche.

I'm still trying to find an engineer that can explain to me why they use Redis for job queuing over Postgres.

Redis for HTML caching: Cool.

Redis for queueing: Are you really sure you need the performance gain? Really? You can do 10k writes per second with Postgres (100k per second with COPY). If you need more than that fine, but for 99% of web apps out there it's just one more moving piece that probably doesn't get included in your backups.

I was looking at that choice for my latest project and the gem recommended (resque?) only using Redis for the job queuing at the benefit of much higher performance over something like delayed_jobs using postgresql in jobs. It didn't seem worth it as job queueing was not the main feature of the server. I suppose if one were doing some sort of real time feature where you had to queue things off a Rails API response thread it would make sense but wasn't doing that for now. But just judging by the number of downloads, the Redis based job queue is possibly more popular for some reason.

Proper queuing in RDBMSs can be tricky.

2ndQuadrant has a good write up of what is required to make sure you don't fall into any traps: https://blog.2ndquadrant.com/what-is-select-skip-locked-for-...

Now if we're just talking about persisting data to somewhere - postgres COPY is pretty amazing.

Spark is not a buzzword if you do analytics, it's the preferred platform, especially over native Hadoop. Honestly most developers have no idea what the heck Spark is or how to use it. NoSQL/Kafka yes, Analytics/Machine Learning is still far too complicated for most to stand up on their own.

Kafka was useful in a project because of the semantics for what amounts to writing to file from network sockets.

Spark is a terribly inefficient solution to any known/stable data processing or analytics jobs. If you want a common format to trade with buddies, it's useful now. I expect something else will come along to replace that fad tech.

>Spark is a terribly inefficient solution to any known/stable data processing or analytics jobs.

Can you expand on that?

Spark is a terribly inefficient solution to any known/stable data processing or analytics jobs.

You gave a typo there.

Spark is a terribly inefficient solution to any known/stable data processing or analytics jobs I have ever come across.

There, fixed that for you.

I don't think that's fixed, because it's not what I assert.

Any job that is predictable, can be done faster and cheaper with some C (or Go) and ad-hoc delegation to load balanced VMs. Spark is difficult to optimize for consistent processes, poor on resource usage, and requires specialized knowledge. Just pay for someone who has done a little embedded programming and stop creating buzzword jobs because a prototype went to production without comparison.

Spark has a place: at large scale. For 100s of GB to a few TB of data PostgreSQL works very well. At least, it does for my team. I don't want Spark, Kafka, NoSQL or any other modern fad near my team's data. It's just not appropriate.

A niche?

If you get deep enough into the hipster development movement (or anything) you lose sight of everything that's going on around you.

That niche being "anyone who needs to store structured data and reliably retrieve it". It's a small market, but there's demand there.

Reply on news.ycombinator.com