Hacker News Reader

Comment by belZaah | Hacker News Reader

I was architecture team lead at Skype 2005-2011 and persistent queues were the only ones around. Basically, because they knew how to scale databases, our DBA team (Hannu Krosing comes to mind) built queues into and on top of Postgres. It happened not just at Skype, too: eBay guys (Dan Prichett in particular) had built a very cool Oracle-based queue solution. Scaling persistent queues seems to be something that needs to be reinvented periodically. Maybe it’s too nuanced of a problem?

View on HN

I actually worked on security for Skype in 2005/2006. And I knew Dan at eBay too.

Using databases for queues isn't a new idea, but a couple things make it different now. One is that I think this is the first time a solution has been open-sourced (not 100% sure on that).

And two, Postgres added SKIP LOCKED in 9.5 (around 2016), which was the performance unlock to make this work without needing a whole DBA team like Skype had.

is this referring to Dan Tarman?

GP wrote Dan Prichett.

Skytools/PgQ was amazing technology that made clever use of Postgres MVCC to provide a high performance transactional message queue. But yes, it was not low-touch to administer.

This article resonated with my experience building what was essentially a distributed task queue using Redis+PostgreSQL with Python workers in Kubernetes. It seems like these systems naturally evolve different patterns based on their specific use cases. The logic of our queue was intertwined with a rule engine. I wrote about building a rule engine here [1]. Another difference to this article is that it did not report back to the client as the events were delivered via web hooks.

There are some different approaches here and there which come from making it application specific, e.g. we added a periodic reconciliation check. I also built a debouncer into the queue to give special treatment to burst in the load.

[1] https://blog.benediktsvogler.com/blog/building-a-distributed...

I know from firsthand experience that there were investment banks doing this at least as early as the mid 1990s - on top of Sybase in the particular case I have in mind. Perhaps not the best performance, but it was trivially easy to inspect, and you could integrate with other database features, for example stored procedures.

Back when I worked for France Telecom circa 1996–99, Sybase Replication Server was the fastest message queue we tested for transporting billing events, by a wide margin, in a way that predated Postgres logical replication by 3 decades.

FWIW, SQL Server had built in queuing support from ~2005. The feature is called Service Broker (https://learn.microsoft.com/en-us/sql/database-engine/servic...)

Much longer than that, the legendary Jim Gray developed MSMQ aka Falcon for them.

Reply on news.ycombinator.com