Hacker News Reader

Comment by zkirill | Hacker News Reader

I think that the job of a CTO is to minimize the surface area of "built in-house" so that your team can focus on things for which your customers actually pay you.

Sure, PostgreSQL can be used for sessions, but Redis has solved this problem on virtually every single platform a long time ago. Your customer probably doesn't even know what a session is, but they will definitely learn about them when your in-house implementation inevitably encounters an edge case.

Of course, PostreSQL has wonderful search capabilities but it was never intended to be used as a search engine. Solr was created in 2004 and is still being improved and used in production every day. Do you know what will happen to your full-text search when your customer types in a mix of English and Chinese characters?

Yes, PostgreSQL addition of SKIP LOCKED was neat but AFAIK the author of that feature himself recommended using a traditional job queue unless you had a very good reason against it. RabbitMQ was designed as a message queue and when you read the documentation you realize that they had encountered virtually every single problem and figured out a way to deal with it so that you don't have to. The documentation pretty much tells you what problems you are going to have later so that you can plan for them today.

Choose boring technology (tm), follow industry best practices, and enable your team to get their work done using the right tools for the job, so they can deliver the product to the customer, and leave the office on time.

View on HN

I get where you're coming from, but I think that folks often underestimate the maintenance burden of additional components in their system.

Sure, if your use case requires it, use Redis for sessions or RabbitMQ for queues. But you can also use a library with postgres, or even write the 40 lines of code yourself.

Each component has its own debugging requirements and tooling. Each component adds a bunch of complexity. Sometimes it's worth it, sometimes it's not. It's not as clear-cut as you're making it be. There are pros and cons to both alternatives.

I actually had this debate at work a few years ago. We needed a queue system and I wrote a postgres prototype in about 100 lines of code that would have worked fine for our use-case and would have likely scaled with us for years of growth.

My boss said he just wanted to install the best thing and be done with it forever. So we ended up spending 10x the time and wrote like 10x the code to integrate a third-party solution and we only ended up using the most basic features. Not to mention additional infrastructure.

I think there exists a midway, where you rely on the heavily-tested open source solution that implements what you want, and a sprinkle more. RabbitMQ for example is that layer that your 40 lines of code would do, but pays off dividends in good nights' sleep.

RabbitMQ is not free. You need to learn it, configure it, understand it, interface towards it, install it, upgrade it. Lots and lots of things.

So those 1000 lines of code (your prototype was 100 lines and this was 10x that) would still have been a week tops, right? That doesn't seem too bad a price to pay for scaling potential, even if in hindsight it wasn't needed. You can't always know in advance which features you are going to need and which ones are superfluous.

If what you need is a simple work queue, you really need to scale very far away (in the magnitude of several thousands of requests per second) to overgrow a solution built on top of PostgreSQL.

People underestimate how bad databases are at implementing queuing systems. I’ve helped replace such systems multiple times.

But it doesn’t need to require 10x the time or code. That just sounds like overengineering. Database-as-queue vs. 10x ultimate solution is a false dichotomy.

Yeah they suck at it. There are so many good queuing systems, and if you’re on any cloud there’s gunna be a cheap managed queue that is super reliable and near zero effort. If you get beyond trivial scale just use a managed queue.

> how bad databases are at implementing queuing systems.

What difficulties did you meet?

People overestimate their requirements, because almost nobody had ANY idea what computers are capable of, and that big tech is HUUUUGE compared to the rest.

You are not Google and you are not Steve Jobs

In other words: boss has no idea, boss doesn’t trust your technical nor business skills

Conclusion: abandon this company.. you’re probably just a cost center and nagging headache in his eyes. If you’re not valued or respected, there’s only way way up, and that’s out

It takes effectively zero effort to maintain my redis instance in AWS, so I’m not sure you’ve got a real argument here…

Perhaps you’re not using it in “clever” ways, as recommended in TFA?

Now you have to pay for 2 services instead of one (elasticache is not cheap), and you may need to account for two differently configured redis instances (setting it as cache store requires a different configuration than setting it as a job queue).

You'll also need to code for two integrations (orm + whatever you're using redis with), which may be a solved problem or not, depending of your stack. And even then, still more complex than just postgres, and more error prone considering you'll either have to ignore enqueue reliability, or find a complex way around it.

Have you considered you’re just a shitty dev?

This is all fine and dandy but the article has a great point here. Redis is absolutely amazing, but if you bring it in you have to care about more stuff. Mo' stuff, mo' problems as they say. You now need to synchronize writes between your Redis and your DB (hello "after_commit" hooks and similar, how is your read-after-write doing?). You need to install metrics for Redis (lest you find that suddenly your application spends a huge amount of time in MGETs or blocks on set operations). You need to have good failover in place at AWS and make sure you do not save anything non-transient into it – yes, this is how Redis is supposed to be used for transient stuff, but are you positive your application can cold-start well enough with a blank Redis? Oh, and now everyone on the team needs to run a Redis locally, and a matching version at that - hello docker-compose...

Brief: yes, a specific datastore is usually better fit for the job, except that until your app requires more performance than Postgres can deliver if you already _have_ Postgres you might as well just stick to it.

Same for SQS - SQS is incredibly performant, but there is a whole lot of features it does not have which a PG-based queue system like good_job will give you out of the box. Just off the top of my head - priorities, separate queues, scheduling - and, do not forget, atomicity with your main transactional workload.

So while it is usually - when everything works great and the workload fits - not a big deal to run a specialized store, it can be more economical and simpler to just stuff everything into the DB until you outgrow that.

> Mo' stuff, mo' problems as they say.

This is basically not true, is my point. There is no meaningful "problem" with throwing up a Redis instance in AWS, this just doesn't mesh with my experienced reality.

Chiming in to concur. Redis is amazing and simple software. You can use a managed service like Elasticache or install the binary on a VM instance. Folks using a relational database for a job queue or a cache when an infinitely simpler and more appropriate tool is available are just making poor technical decisions.

But it's not simpler when you consider all the things you've got to do around and after installing that binary on a VM instance. Consider the overhead of managing it - monitoring it, updating it.

Failover when the node dies. Clustering for high availability?

Backups? For a cache, probably not, for a job queue broker, probably necessary.

Making sure your app deals with inserting into Redis on successful transactions and not when a transaction is rolled back.

Getting up and running can be fairly painless, staying running on all edge cases and handling partial failures is what gets you.

I’m confused the scenario you have where a) a singular postgres install which does everything is acceptable vs b) as soon as redis comes into the picture, suddenly you need HA and monitoring and apparently running transactions with full ACID integrity?

It’s just a nonsensical and unfair comparison. You can run a single Redis instance with normal rdb disk syncs and don’t ever update it for years on end without issue. Is that guaranteed resilient? Absolutely not, but that’s not the scenario in discussion. We’re talking about the context of a bootstrap/MVP scenario, not an enterprise setup.

I’d take a single-node redis job queue everytime over a HA citus/postgres cluster improperly acting as a queue.

I think the point is they have a Postgres server running anyway as the datastore and the job queue being in Postgres gives you HA, backup and Transaction for free. I think Redis in particular won't give you transaction right?

Needing Transactional semantics for jobs alongside an application operation makes a lot of simpler queue/tool choices difficult.

Thank you for the nuanced assessment, I would tend to disagree still.

I agree with the OP's point of the overhead of adding more things to maintain.

For Redis though, the overhead is far more trivial than something like Kubernetes or Kafka, or even Elasticsearch or MongoDB

The overhead is actually the conversation we're having right now about whether postgres or Redis is better. It's not that postgres is hard to use or less perfomant, but that there's memory overhead in "Here's how you use Postgres for session management, here's how you use it for application building". Use Redis for this, Postgres for that is easier to grok.

That's not the scenario they're describing, postgres has most likely already been designed and worked on to scale to their workload, using postgres means you don't have to replicate that for another system.

Speaking from experience, you can get away with not understanding Redis for a while. Then one day you'll wake up and everything will be on fire, because you used Redis wrong, and now your main and replica are in a death spiral DoS'ing each other trying to pass a 1TB replication log back and forth.

You don't need to learn your tools for doing simple stuff under normal circumstances. You need to learn them to do bespoke surgery on live data while everything is on fire and the customers are threatening to fire your company. Or better yet, so that you can avoid doing that altogether by anticipating limitations during design.

That being said, doing everything in Postgres is also going to bite you if you have moderate scale. This is really the same mistake again. Postgres looks like a big truck you can just load up and load up, until you wake up one day and there are cascading failures across all services that touch that database, because you wrote a dumb query that took a lock for 5 entire minutes while it did an HTTP request. It's robustness will lull you into thinking something is working well when it's actually barely working.

(Before you object, yes, it is a better idea not to have multiple services talk to the same database, I hear you. And no, you shouldn't ever hold a database lock while doing an HTTP request, believe me I know. These things can happen.)

Er, I really feel like you’re not understanding how simple AWS makes managing a Redis instance.

I’ve been using Redis for nearly 10 years and it’s been a seamless and pleasant experience. Honestly it sounds like you’re taking your specific experiences and overgeneralizing.

Partly I'm sharing war stories because that's a fun thing to do, I'm not being entirely serious. Partly it's that you said that wasn't your experience of reality, so I broke off a little piece of mine and offered it to you. I'd suggest we're both generalizing, and as long as it's not taken too seriously, that's fine; it's how shop talk works.

I don't know the nature of the applications you've been working on those last 10 years, but it was more or less the main database for a high bandwidth, low latency service I was working on, also using Elasticache.

Problem spaces vary. If you're using it as a cache with modest load and consistency requirements, maybe you never need to understand it. But those sorts of requirements often creep & change out from under you.

So if you're saying, Elasticache did a good job of abstracting Redis, sure, I agree. If you're saying, there is no additional cognitive load to adopting a new service in your data path, because you don't even need to understand it - that puts a shiver up my spine, and makes me hear Pagerduty alerts in my head.

Keep in mind the context of this thread. It's some shitty blog suggesting you do every fuck thing on postgres. Myself and the other guy are simply suggesting that running a single-node instance of redis is an infinitely better and simpler choice than implementing a cache or a job queue on a rdbms.

I don't feel designing for a guaranteed high-availability application was part of the discussion at all.

If you don't find it applicable, feel free to discard what I'm saying, I take no offense. But I just don't quite understand your perspective. Maybe I have oncall firefighter brain rot, but a distributed job queue is exactly the sort of thing I'd want to be available, and a cache is something I regard as being very dangerous and requiring utmost care.

You use SQS, couple clicks and you’re done. You shouldn’t be using a cache for a queue. You use a queue for a queue.

Not disagreeing, but purely for interest, Redis contains a queue-like primitive.

https://redis.io/docs/data-types/streams/

You can make good job queues out of this, combined with sharding or consistent hashing, for low(ish) latency applications. Each shard has a stream, they operate on data stored in Redis, and you pass them the key to this data over their stream.

But SQS is great, and a great rebuttal to the article. Totally easier to prototype a job queue that way than with pg, and you probably won't need to move off of it.

Sure, but you still need to run redis vs click a couple of buttons and create a queue, offloading the entire management to AWS staff.

I have used both SQS and PG-based queues and for the smaller workloads/smaller systems (read: "not very very large systems") I now prefer the latter. There is also a non-trivial amount of stuff that we turned out to need for operating SQS at scale on the application side, basically to compensate for the things SQS does not have. It is great it doesn't have those things, but if you have a smaller application you might want to have those things instead and sacrifice a bit of scalability.

The advantage being that you could sort things to implement priorities and such? Did you use listen()/notify() at all?

ETA: seeing your list of missing features now, that all makes a lot of sense. In my mind the biggest advantage of SQS is that it glues together all the other AWS offerings, so you can go AWS -> Lambda for an instant job queue (with concurrency limits, etc. so you don't blow your hand off - perhaps undermining the simplicity argument). But everything you're saying makes sense if your job queue needs any degree of sophistication.

Redis isn’t just a cache. That’s memcached. Also SQS absolutely sucks for a job queue as soon as you want to do anything like control concurrency or have job priorities, but if your needs are simply “I need a background job queue” then SQS is likely a great choice.

Yes, mostly. There are a few things to take into account though:

- No multiple queues - No priorities - Practically no scheduling (the delay is very limited) - Creating and tearing down a queue takes a lot of time and the number of queues is subject to AWS account limits - The FIFO/LIFO semantics (remember about no priorities?) will bite you when you least expect

It does have great durability unlike Redis though and will scale to much, much larger queues in an easier way.

As long as it works the first time and everyone on the team is fine installing a local Redis - there is very little problem. If the code doesn't make assumptions about read-after-write consistency for jobs. There will be "problems" (or - rather - things you will find out you haven't accounted for) when, for example, an improper URL is used and your Redis fails over. Or you do not have a replica configured (someone decided to "let's save some budget of team XYZ and is this really necessary it is transient after all"). Or you started using something that saturates your Redis. Or that you haven't configured alerting on Redis metrics...

It's all normal stuff, by a long shot not the end of the world, but it is stuff that you need to do, and it is more stuff, and it can bite you if you come unprepared and "just clicked a few instances into existence last year".

Every added service also gets multiplied by the number of environments you need. Sure, prod is just one. But you need a staging env too. And CI needs the service to run the tests. And you probably want a feature-branch environment. And local development of course. It adds up and every service you add gets multiplied by the number of environments.

And each developer has their own dev env. QA’s may also run things locally.

And then there is monitoring, dealing with security issues, and upgrading.

It all multiplies.

Elsewhere people are debating whether Redis adds that much more overhead. I think they are missing the wider point. It’s about how many pieces you will have. 1 is simpler than 2. 2 is simpler than 3. 3 is simpler than 12.

Nowadays there are so many great specialized tools that do certain things really well. And for most use cases, you don’t need them. And when a time comes that you do start needed them, you add them then. And people will grumble about the idiot who implemented full text search inside of Postgres, while being completely blind about the 7 years of saved time NOT managing elasticsearch across 38 environments and quarterly upgrades.

SaaS is nice, but you still have to worry about Redis drivers, local development set-ups, etc. And unless you can make do with AWS’ free tier, that Redis instance isn’t free.

This doesn't apply to everything. But yeah redis does what it does really well. I almost feel like I want to spin redis up before I spin a DB up. It is excellent tech. Can use it for caching. Sessions. Distributed locks (niche but when you need it it's fantastic).

But yeah I think this is more applied to the folks who decide to spin up a large service when they just need a messaging layer that their DB can solve for them. It has non insignificant cost.

Redis is pretty low maintenance tho in many configs (k8s, on a vm, or a service like on aws have all basically been zero maintenance for me).

Those 40 lines of codes give me nightmare. What if the dev disappear and we know only the C binary, source disappeared ?

Little tiny custom stop gaps are the worst in a bit system.

> Choose boring technology (tm)

I'd call Postgres pretty solidly "boring technology", including for session storage and job queues. People were storing sessions in SQL databases when I got my start in 2005!

It won't address every scale and every use case, but then, that's never your project's requirement anyway.

I frequently see the term "boring technology" treated as a euphemism for "what I'm accustomed to".

Absolutely on point.

Postgres can and will scale to workloads that average developers can't comprehend. I've used Postgres professionally since 2003 doing everything from run of the mill web work to high volume log aggregation systems with massive datasets. Postgres can ingest and index hundreds of thousands of records per second on pretty modest hardware by today's standards. One just needs to learn the tool.

Postgres is far more "boring" and battle tested than Redis. All the hipster tech that came out of Web 2.0 companies (including Cassandra and Redis) has hallmarks of being built by junior developers who refused to learn and then build upon state of the art and went on to reinvent many wheels quite poorly.

You had some reasonable points but then you segued into name calling and baseless negativity. Both redis and Cassandra are well built, well understood common industry tools that have been use at scale for many years now. With their own trade offs yes , but quit it with the “built by junior hipster developers” BS, it’s not a good look

I stand by my point. It was all reactionary stuff. I went to the conferences and saw the talks by those bright eyed young developers. Their disdain for RDBMS and SQL was as fervent as it was misguided. They hailed a new era of Big Data and NoSQL. Hadoop was gonna be the way to store it all or you weren't webscale.

Forward a decade and most of it is in the dustbin of unmanageable tech while good ole RDBMS outlived them all. Mongo is about the only one I still hear about and see pop up in job ads. Might share the fate of Hadoop too when people learn that Postgres can index JSON too and even be sharded if you need to (you likely don't).

If you think that redis and Cassandra are in the “dustbin” then I don’t know what to tell you, except that you’re possibly very out of touch. Especially redis, it seems to be used everywhere

You may be confusing redis with mongo which definitely suffered that. Redis is simple and well designed when used as directed.

Won't adding item to the queue leas to fsyncs on each commit? That first thing comes to my mind and worries me

Fsyncs can be combined. You don‘t need 40k fsyncs for 40k inserts. While one fsync is running multiple inserts are queued and fsnced together when e.g. the last fsync is done.

Counter argument: 1 postgres instance is far easier to manage than a slew of servers. I can spin up a heroku environment with a postgres very quickly, and with little effort.

If you're starting out with a team of say 4, and that team will stay small, don't waste your time integrating complex tech.

Solr is DEFINITELY better than postgres fulltext search. However pg_search and a clever index has let me solve 95% of all our search needs in almost no time with zero maintenance. However once you want to get into search complexities, Solr quickly outperforms postgres, but you have a lot of setup and env management to do now.

So it is all about team size. My instinct is at around 10 engineers you should start thinking about which complexity is worth pulling out of postgres and into its own service based on needs.

Solr is better in snots use cases but does things like sparse indexing very poorly and Postgres is actually worlds better

"Snots use cases"? Is that an unusual autocorrection for COTS?

I very much disagree with the idea that a CTOs job is to minimize code built in-house.

Every app has different needs and there's always some code that would be a lot simpler if it were written in-house and specifically built for the needs of the application.

Abstraction has a cost and if you take everything off the shelf you'll end up with a much higher overall level of abstraction in your codebase. Plus, your engineers aren't going to understand third-party code as well as in-house code.

I've worked at companies that wrote almost everything in-house and I've worked at companies that had a phobia of in-house code. Both had problems and I think the real solution is somewhere in the middle.

Edit: I also get a "nobody was fired for buying IBM" vibe from this.

Not only that, but in-house offerings can be a competitive advantage. Sometimes it's over-engineering, and sometimes it's the secret to what makes your business successful over competitiors.

In house has the ability to build only what is really needed and nothing more, and can adapt to your specific needs, it can also identify unique to your domain challenges and tailor solutions specific to that.

A good CTO has good intuition into when and what makes sense to invest in an in-house solution and what is best using a self-managed open source solution, and what is best using a paid managed offering, and all manners of hybrids.

Honestly the biggest competitive advantage isn't in the thing you're building in-house, it's just the fact that you can debug it 10x faster than your competition.

I never understood this until I saw some dysfunctional enterprise companies. I've seen entire teams spend 3 months doing fuck all because they can't get a local dev environment or build pipeline to work.

If you only have Postgres, and you're weighing up whether to add a second dependency instead of installing a Postgres plugin or writing 50 lines of code, of course it's going to seem like an obvious choice ("I can get this working right now instead of spending a day or two on it! Wow!").

If you make a habit of making those decisions again and again, then your project is going to spiral out of control before you even realise it.

I think this may underplay the additional operational cost and risk in deploying multiple classes of services.

You added RabbitMQ for that one queue use case? You suddenly need to handle some health check edge case in prod since your programmers doesn’t have experience with it. Just added redis? You now have an extra set of server and client sdks to regularly patch up.

Etc. Sure, there’s a point where it’s logical to add a new class of services, but it’s not remotely close to zero (which is how I read the comment).

The enemy's gate is down. If a CTO brings the surface area of "built in-house" down to zero while accomplishing all of their objectives, ad infinitum, they win the game.

Obviously, it would be impossible to maintain such an advantage in real life for any extended period of time. However, orienting your team towards that goal gives the CTO a way to quantify risks and costs associated with accomplishing their objectives. Health checks and SDKs are standardized commodities that can be implemented and maintained at a predefined market rate that's always approaching zero. Finding and fixing a bug in your proprietary code has a potentially infinite cost.

Finding and fixing a bug in your own code that interfaces with a Kafka cluster that's tripping a disturbuted systems edge case is infinite squared. Solving it requires a huge volume of knowledge about complicated systems and topics as well as debugging across system and server boundaries.

It's all trade-offs. Complexity is complexity, whether the code is yours or not.

Thinking about software costs in this way is so oversimplified that it ends up being wrong in practice. I'm reminded of the McNamara Fallacy.

Also, SDKs are not standardized. Third-party software still gets updated which means code maintenance for your team. And, even worse, it's software that is generally opaque because, by definition, it wasn't written in-house. I worked at a place where they had a phobia of in-house code and the end result was huge amounts of time wasted on updating libraries and debugging issues caused because we didn't really understand the code we were using.

The real answer is deeply understanding your product and finding the right mixture of in-house and third-party software that maximizes simplicity while also allowing for flexibility and growth in ways that matter for your specific product.

Good reply, thanks.

Now your engineers need to understand all these technologies instead of just postgres. As usual the correct answer to when to repurpose existing tech in your stack and when to add new tech depends on your specific needs, team size, etc.

The majority of engineers understand basic SELECT and INSERT semantics. Correctly building queuing and caching systems on top of database concurrency primitives is an order of magnitude harder than just using RabbitMQ and Redis.

> I think that the job of a CTO is to minimize the surface area of "built in-house" so that your team can focus on things for which your customers actually pay you.

The "built in-house" things are the only things that actually make your company competitive and provide shareholder value. There is no value in downloading and installing commonly-available stuff from the internet.

Which is exactly why if you aren’t a database vendor, you shouldn’t have your engineers spend their time maintaining a proprietary database technology.

If you maintaining a proprietary database technology solves a problem that provides some shareholder value, then go ahead and do it.

Maintaining off the shelf software that you got from Github provides no shareholder value whatsoever, however. It is a pure cost center.

As a CTO you do not want to be the guy that runs a pure cost center department.

There is negative value in rebuilding things that could have been downloaded for free from the internet. Postgres, MySql, Sqlite, RabbitMQ, Redis, Kafka and all the other common tools each have had thousands upon thousands of hours sunk into bugfixes, correctness guarantees and performance work. Rebuilding a poor version of that does not provide shareholder value at all.

Many others have said this elsewhere in this thread, but the cost of using all those things is not free. Every added service is an extra burden to maintain, all the way from writing code through to running and maintained in production.

Sometimes, the cost (both immediately and long term) of introducing a new component into your stack will exceed the cost of building and maintaining an inferior in-house solution, and sometimes it won't. Either way, the cost of these freely downloadable services is not just their cost to download.

The problem isn't using one of these things, the problem happens when you use each of these things. Now you have a distributed system where data is in different systems and you have data inconsistencies or have to deal with distributed transactions. The advantage to using postgres is you can continue to use a single transaction for data integrity.

I’m not sure I follow. Postgres has been around forever, it’s basically what I would call boring technology. Sure they’re still innovating, but the core tech is mature. An example of a “not boring” database would be like Cockroach or Planetscale.

I think "use postgres for everything" is a good solid foundation to start with (if you dont have a really good reason not to and know the exact use case beforehand). Can redis, rabbitmq or solr etc do a better job for a specific user case, of course becasue they are dedicated tools for the job. But the question is, is the builtin tool in postgres "good enough"? And for many thing they are.

I see postgres like a tractor. It's the most important machine and you can attach alot of different equipment's to get alot of jobs done. Some times you notice a task need a specific machine that the tractor is not suited for, then you invest in that machine. To invest in all specialized machines from the beginning just because they can do one single job better than the tractor is not an efficient way of farming, there is a price to have all those machines too, that does not necessarily result in a better outcome.

Redis doesn’t “solve” sessions.

This is the usual debate on "hammers and nails".

You are absolutely right, but what if(just an example) you need transaction behavior between your queue system and your database? Good luck with 2pc or using saga and similar.

Or, as other people stated, how much will cost to maintain postgres + redis + rabbitmq vs only postgres?

In my opinion, the golden rule is: *use the most general purpose tool you know unless you hit a hard limit of the tool, in that case start moving to a specialized tool*.

Couldn’t say the first sentence any better. If your company doesn’t make money directly from the “thing you built” than it’s losing money, lot’s of money.

This is build vs buy vs run. Yes, what you're saying might mean slightly less is built, but there's a lot more to run. Having ops people who know Postgres inside out might go better than having ops people who have to try and know four totally different technologies a bit.

Building on this, for many companies, the leader of the IT org has the main responsibility of focusing on end (or outside) customer needs at the highest quality with a low "Total Cost of Ownership".

Part of being a CTO is knowing when that specialization, both in tools and in-house development, make sense for your context.

Choose PostgreSQL as long as possible.

This is probably good enough for 99% of things out there running stuff.

For everything else you have architects which will do the right thing for you.

What is a session?

Reply on news.ycombinator.com