- You’ll want writers to create a temp file and then rename it into place; otherwise your reader might pick up a file that’s only half-written. (POSIX rename() is atomic within a filesystem.)
- It’s also helpful to have an `in_process` folder, and a cronjob to send alerts if a task sits in that place too long. That way you can quickly catch crashes and other failures.
- You can have multiple readers; they cooperate by rename()ing the input file into in_process/ before they start working on it, and ignoring ENOENT (which indicates some other process got to it first).
- This pattern is great for loosely-coupled systems. In my case I was importing orders into a system that ran very slowly; once a reliable import queue was available new use cases kept presenting themselves, and it was very easy to add them; since the API was just “write a file in the correct format”, individual scripts could be written in whatever way was easiest for the particular task at hand. (It helped that the input format hadn’t changed in a couple decades; if it had needed changing a lot you’d want a more structured system. But this is the same as any distributed systems interface design.)
I wonder if the system is still in place. Last time I checked the Jenkins folder was occupying 270 GB (!) of the 10 TB shared FS, most likely because FS block size was 1 MB.
On the topic of message queues, I wonder if anyone ever designed a mechanical, asynchronous hardware IPC mechanism based on tape. Imagine: an endless-loop tape that stretches between a "sender" tape-drive on one machine, and a "receiver" tape-drive on another machine; with each tape drive locally tensioning the tape as it passes through, but letting it hang slack outside the drive; with two tape-width buffer boxes in between the two drives, for taking up the slack; and where each tape-drive has a strain gauge on its input-end capstan.
With this design, you'd never need to seek the tape (backward, at least) — the receive head would just read the tape forward at whatever speed it could until it ran out of slack in the tape (i.e. blocked on read); and the send head would write the tape forward at whatever speed it could until it ran out of slack (i.e. blocked on write.) The tape itself would be a literal, physical ring buffer.
I wrote some VBScript that wrote a locking folder into a sub-folder in a network folder, then launched VNC targeting the proper VM. When VNC was closed, the script would complete and delete the locking folder. That let us know what was available and when. I always thought it was hacky, but at ~60 lines it's incredibly simple and has never really failed.