Anything can be a message queue if you use it wrongly enough

This is an important lesson - if someone is using your tool in unexpected ways don’t just shut them down; there’s likely a business case that could be identified and specialized in.

Did you detect this MixPanel or something?

It's amazing with how much creativity users will abuse your creations. ;) And in many cases, something new is born out of it. The problem is getting the information about just how people are using your service differently than you've intended. Sometimes it's impossible to tell from traces, analytics or logfiles alone. But finding out can be quite an advantage, especially if you're a startup probing for PMF. The best thing that can happen is that you have a channel to your customers that's constantly open. At WunderGraph, we use Slack for that, and it considerably lowers the barrier to just check in and have a quick discussion. The sooner you find out about use cases, the better - and ideally create a product around it. :)

This isn't theoretical; many companies do PostgreSQL async 1:N physical replication, by using e.g. https://pgbackrest.org/ to have the primary push WAL segment files (a.k.a. "the last n milliseconds of packets" in the write-ahead log) as objects to S3. All the read-replicas then independently discover the new objects in S3 as they become available; fetch them; and replay them.

> You could do even better if you out-of-band signal the readiness so you do not need to poll while idle.

S3 and its clones have "object lifecycle notifications", where you can be informed by a push-based mechanism whenever a new object is put into the bucket.

But — what do you have to do, to get these notifications?

Subscribe to a real message queue, that S3 puts these notifications into.

So using it here would be somewhat cheating ;)

Try it with CloudFlare's S3 equivalent offering

Why not sqlite with a lockfile? /s

Segment did so quite successfully https://segment.com/blog/introducing-centrifuge/

I am doing the same with SQL Server. The messages table is more of a bus than a queue in our case (columns like ReplyToId, etc). Using it for RPC communication between cloud bits. Much cheaper than Azure Service Bus and friends.

This is actually pretty common, and usually a "good enough" solution. You can also add things like scheduling (add a run_at column), at least once execution (mark a row when it is being processed, delete it only when successful), topics, etc with minor modifications to your table.

If you want something that works "well enough" I'd say it's a reasonable choice.

See also: postgres with SKIP LOCKED.

Great talk on it from Citus Con 2022:

Queues in PostgreSQL | Citus Con: An Event for Postgres 2022 -- https://www.youtube.com/watch?v=WIRy1Ws47ic&list=PLlrxD0Htie...)

Citus is a horizontal scale out extension for postgres -- imagine the power of those two things together!

JK you don't have to imagine:

https://www.citusdata.com/blog/2018/01/24/citus-and-pg-partm...

I briefly worked for a major corporation 15 years ago that did this with SQL Server to create distributed worker processes to handle all the AI-generated used car listings and photo recolorings [0] for almost all of the used car lots in the country.

[0] Why take hundreds of photos of Honda Civics in red, green, blue, and black when you already have a dozen in white?

I’ve run into all sorts of database locking issues and concurrency issues when using a database as a queue. I saw that mistake made a long time ago and I would never do it myself.

This has been a thing since before databases were relational. 4G languages (Progress, etc.) were especially nice for their ability to wrap a queue table around a series of reversible transactions, if you coded things right .. meaning a lot of modules written for app infrastructure were based on an 'inbox table' methodology ..

I've built a hybrid task queue/process supervisor on top of SQL. Classical task queues like Celery didn't exactly fit our use case: a single process could run for hours or days, but in case of a node failing, it must be resurrected elsewhere as soon as possible (within seconds). I didn't have the time to re-architect everything for Kubernetes, or rewrite half the product in Erlang; so I built that weird thing. It's been super stable, running mission critical code, and making us money - for several years now.

Nothing wrong with using the DB as a mq, especially if the load is small enough. Plenty of tools are built on that, these two come to mind

https://github.com/procrastinate-org/procrastinate

https://github.com/bensheldon/good_job

Nuxeo unironically used a db table as a pub sub system between the cluster nodes for cache invalidations.

I implemented a message queue in MySQL too and it worked pretty well. Incoming messages would be written to the table and the workers would poll the database each cron period and process whatever rows were in the queue. To avoid race conditions, the workers would lock the records they were working on and then delete them as soon as the work was complete. It was simple but it worked just fine for my purposes

If you don't want to publish events from uncommitted transactions you'll have to first store them in a local table and then move them to the queue after the commit. But if all consumers have direct access to the database anyway...

Email is a message queue in the most literal of senses, isn't it?

A famous financial news paper used an FTP server as its kafka "fan in" node.

It was so old that the 1u case started to droop in the middle. (it was an sparc something or other with real SCSI ultra 320 drives in them)

Ha! I think i know what company you’re referring to

Active Directory is (was?) capable of running replication traffic (for very particular use cases) over SMTP. I always thought that was ingenious.

Using SMTP to transfer messages seems just plain common sense. Far different than IPv6-over-S3.

I co-founded a webmail provider in '99, and when we needed a message queue reaching for our heavily customized Qmail setup was a relatively natural choice. I've mentioned it here before. Provides all the routing and retries we needed, and made debugging trivial (e-mail from our desktop clients to the queues worked; cc:'ing a real mailbox with copies worked; checking out queues with POP3 worked...)

I always thought using e-mail in this would be good for systems that occasionally need a human in the loop. Normally one service processes and mails results to the next, occasionally something is forwarded to a human, who can make some decision, edit the mail, then forward it along back into the automated path.

Before there was REST and other http based standards ebXML was a hotness and tools like bizspark etc. You could see the logical progression through:

I fill in the order form and post it. They mail back an invoice.

I fax the order form, they fax back an invoice.

My computer sends the order form in a structured way over email.

My computer sends the order form in a structured way over http.

And that's why many large telcos and banks still use ebXML in their B2B transactions. It's fundamentally the same business process and logic it's always been, with glacially slow improvement in performance over time.

One of my customers used email (a gmail account, no less) as a message queue between their front end site and the back-office processor. This worked quite well for close to a decade I think.

It basically evolved from when applications from their original customer facing site were emailed and manually entered into the back-office system by a human. They were looking to automate this with a minimum of changes to too many of the moving parts at once, so I reformatted the sent email to contain an XML payload so the new back-office automation could read and process it (and in a pinch, a human could still review any problem applications) using Java's mail APIs.

Things evolved, the front-end web site got replaced with a Wordpress site, but the email message queue kept working for a long time. In the last year or so it was getting more and more onerous though. Reconciling information between the front-end and the mail box showed not all emails were being delivered, and authentication to gmail was becoming more and more of a burdensome moving target.

I just recently replaced the whole thing with an API call made from the back-office to the Wordpress site to access the stored data. (The original site didn't store, just emailed, which was why this was not an option historically.)

Um... why?

If you have to point out how clever you are, you're probably not as clever as you think you are

Intentionally misleading your readers is exceptionally poor writing.

Spoiler: "we can of course ignore air resistance because chainsaws cut through the air like butter"

I'm going to have to spoil some of it to point out that part of what he's done is reinventing the idea of delay-line memory, although he makes no mention of this: https://en.wikipedia.org/wiki/Delay-line_memory

...of course if that's all there were to it it wouldn't be interesting, but I won't reveal what the rest is as per your suggestions. :)

> Whoever could accomplish a simple task with the most ridiculous system would win.

See also:

* Microservices

* "Web scale"

* Kubernetes

* Cloud computing

I used to host a competition called Anything But Ethernet.

(You could use ethernet hops, they just didn't score points, so say you had some ethernet-to-whatever bridge, only the 'whatever' segment would count.)

Over the years, we saw physical proof that a T1 circuit would in fact run over barbed wire, we had a human-in-the-loop following a DDR-esque stream of arrows to stomp on a pad that encoded data from one hop to the next, we had a writable RFID tag stuck to the front bumper of an R/C car that would drive back and forth between readers...

FYI, this is an old Unixism called a "spool directory" :)

Sometimes a filesystem-based queue like that is all you really need. Recently I used S3 API compatible storage for something similar, as introducing AMQP or pubsub would have been immensely overkill for this low volume component.

This is a very sensible design for small-scale queueing systems. A few implementation notes I’ve learned along the way:

- You’ll want writers to create a temp file and then rename it into place; otherwise your reader might pick up a file that’s only half-written. (POSIX rename() is atomic within a filesystem.)

- It’s also helpful to have an `in_process` folder, and a cronjob to send alerts if a task sits in that place too long. That way you can quickly catch crashes and other failures.

- You can have multiple readers; they cooperate by rename()ing the input file into in_process/ before they start working on it, and ignoring ENOENT (which indicates some other process got to it first).

- This pattern is great for loosely-coupled systems. In my case I was importing orders into a system that ran very slowly; once a reliable import queue was available new use cases kept presenting themselves, and it was very easy to add them; since the API was just “write a file in the correct format”, individual scripts could be written in whatever way was easiest for the particular task at hand. (It helped that the input format hadn’t changed in a couple decades; if it had needed changing a lot you’d want a more structured system. But this is the same as any distributed systems interface design.)

I'm doing the same right now. The incoming messages are coming from a radio network at 1200 baud, so I don't foresee capacity problems :D

At my old company, we had a system where we had a limited number of physical machines that multiple testers would need to remote into. We didn't want to disallow two people logging in simultaneously because sometimes that was necessary, and if someone stayed logged in then wen to lunch, where was no way to kick them off to get access without waling to the server closet, pulling up the machine on the KVM, and kicking them that way.

I wrote some VBScript that wrote a locking folder into a sub-folder in a network folder, then launched VNC targeting the proper VM. When VNC was closed, the script would complete and delete the locking folder. That let us know what was available and when. I always thought it was hacky, but at ~60 lines it's incredibly simple and has never really failed.

tom7 is a national treasure. Someday I'll have an idea as absurd as this.

Really? This is the old internet to you?

> <Cadey> Hello! Thank you for visiting my website. You seem to be using an ad-blocker. I understand why you do this, but I'd really appreciate if it you would turn it off for my website. These ads help pay for running the website and are done by Ethical Ads. I do not receive detailed analytics on the ads and from what I understand neither does Ethical Ads. If you don't want to disable your ad blocker, please consider donating [snip] or sending some extra cash to [snip] or [snip]. It helps fund the website's hosting bills and pay for the expensive technical editor that I use for my longer articles. Thanks and be well!

that's the joke.

I mean. Chain, queue, it's not that different if you squint hard enough...

IPv9 is already a thing

> What could be worse than IPv6?

IPv4?

But seriously, I appreciate that the blog was written using IPv6, not some other old and deprecated legacy tech.

Like any good extended comedy bit, you have to read to the punchline to get it.

I believe they meant "7 times", so around 85%.

I'm pretty sure this is still the case. I'm not sure how cross-platform that assumption is (it won't work in Go where it has heapstacks I don't think), but classically yeah the stack is put is slightly faster memory with fewer access barriers.

By definition, everything on the heap got where it is dynamically, so the minimum number of pointers to chase to find it is 1.

In contrast, what’s where on the stack can often be (and often must be) known statically by the compiler and accessed directly, even moved entirely out of memory and into a register. (It’s possible to do this with suitably constrained dynamic allocations, but the optimization is much harder.)

Many architectures have hardware support for stacks, which could be slightly faster than arbitrary load/stores. Only works in the function owning the stack frame of course, if you pass a pointer to a stack object somewhere else, it's back to being normal memory.

I'm not part of this subculture, but I honesty enjoy this type of idonsyncratic presentation which always was, and hopefully always will be part of hacker culture.

The content is not only relevant and hilarious, but I'd much rather personal blogs like this keep being updated instead of seeing the the sterile Medium or LinkedIn Pulse versions, usually written to impress future employers.

Zany visuals, doing things with your friends just because you can and writing up detailed accounts of it to share with everyone is the quintessential hacker culture. If Hackers' Cyberdelia was set in the 2020s, no doubt Mara would fit in.

I was gonna add Route 53 (to have each node set its own DNS record), but I'm saving that for part 2. Be afraid, part 2 is coming.

My read time estimate code is here: https://github.com/Xe/site/blob/aa3608afa6c62695ca0ab139f823...

I've been trying to play with the constants over the years to make the read time estimate more "accurate", but it's a tough nut to crack in general. So I can go over my numbers more accurately, how long did it take you to read it?

Estimate includes the additional latency you will experience while downloading the post through the mechanism described in the post, which is the only true way to read it.

Wait, are you saying that AWS S3 is turing complete? How the hell if so.

Here's the key paragraph in the article:

> In Linux, you can create a TUN/TAP device to let applications control how network or datagram links work. In essence, it lets you create a file descriptor that you can read packets from and write packets to. As long as you get the packets to their intended destination somehow and get any other packets that come back to the same file descriptor, the implementation isn't relevant. This is how OpenVPN, ZeroTier, FreeLAN, Tinc, Hamachi, WireGuard and Tailscale work: they read packets from the kernel, encrypt them, send them to the destination, decrypt incoming packets, and then write them back into the kernel.

This is one of those articles where the journey is far more interesting than the destination.

(Edit: this comment made more sense when it was replying to a different complaint about the article; the parent comment seems to have been edited in the interim.)

Setup a TUN/TAP device, which is a file one can read (egress packets) / write to (ingress packets). Setup appropriate ip routes.

Read (egress) packets from the TUN/TAP device.

Serialize them, assign sequence numbers, and upload them to S3.

On the recieve side, poll for newer objects (packets) in S3.

Write them to your TUN/TAP device.

---

Here, the TUN device is a router using S3 as gateway.

So the premise of the problem is the need to egress traffic to the internet from a private VPC, without using an expensive NAT gateway. The solution is to do NAT manually by having an EC2 instance in a public VPC, and tunneling traffic from the private VPC through that instance.

The meat of the article is how to create that tunnel using S3 for data transfer instead of using a more traditional VPN service.

my dear child, the fun is not the destination, the fun is the slow decent into madness to get there. For other people: don't read the summary, read the whole thing. its a work of art.

> ...it was simple and it worked – especially for a short-staffed technical team facing tight deadlines and a rapidly increasing user base.

> For four years, the database message queue formed the backbone of DigitalOcean’s technology stack.

The lesson I got from that article is that a database messaging system got them very far and was simple.