Is there a reason why there aren't 2 types of jobs? One for each stage? You mentioned the first stage there is some work performed, then it sounds like you need to pass on the work again to another worker for the second stage.
114 karma · joined August 22, 2016
The one feature which may not be obvious yet is the ability for workers to mark a job successfully completed or failed with a result and then retrieve it later. The workflow looks like this:
* Client A: Backgrounds a Job A
* Client A: Backgrounds a Job B
* Client A: Backgrounds a Job C
----
* Worker A: Picks up Job A + Completes
* Worker B: Picks up Job B + Completes
* Worker C: Picks up Job C + Completes
----
* Client A: Picks up the result for Job A,B,C.
This allows a single client to concurrently process multiple jobs within a single process and retrieve its result. This is what I like to call "Gearman mode"[0], as it was modeled after that project also. Useful in languages that do not have well defined concurrency. This is a niche use case and may not be needed by everyone, but very useful as soon as you need it. This will become more obvious when I have clients for these languages.
Lastly there are some subtle enhancements such as retry support and synchronous processing (submit and wait for result).
Thanks for the great question. This is a very popular question and I will FAQ it.
There is no way to define the dependency automatically, the workers can create any type of workflow though, including if Job X fails.
Workq is built on the higher level concept of a job so the feature set is refined around what a job is. In Workq, a job must successfully complete or fail, and optionally a result passed back to the client. A job can be retried when it has timed out or even when it has explicitly failed outright (maybe there was an temporary error with your API provider..etc). You can say: retry the job if it has timed out up to 5x BUT let it explicitly fail only once. These small refinements help streamlined the concept of processing a job fully, not just a blob of message.
Also there are some other key things such as job scheduling based on time which don't exist in RabbitMQ, but are usually offered in libraries such as DelayedJob..etc.
I spent 80% of time on Workq just writing tests for it and there is still so much to account for even as a standalone system.
On the bright side, projects like etcd & consul (I believe the author of Raft is helping out there) are getting better and better and can be embedded.
As for the HTTP2 portion, the server details would be abstracted out especially with HTTP2 support in Go 1.6. At the time I looked, HTTP2 clients for various languages were still popping up and stabilizing (I think they still are). I didn't want that to be a factor when I was developing clients outside of Go (for example PHP). In addition, an important goal was to develop extremely small clients, where I understood exactly what was going over the wire.
If there are enough direct tooling benefits that HTTP2 can offer, it would be fun to experiment with it as an alternative interface. Funny enough, the first prototype name of the project was "httpq".
The ping example was just given so I could enqueue a job named “ping” as a client and respond with a “pong” text result from a worker. Just a silly example :). Real use cases would be background jobs such as sending emails, http downloads, image resizing…etc.
From my own experience, the text commands are easier to test against, especially its boundaries (inputs to the server) which makes client development significantly easier.
There will be some sort of interface for the storage. I'll keep in mind pluggability. It is something Gearman had back in the day also[0]. Most likely it will be persistence to disk for some time and once more clarity comes out, possibly pluggability.
Internally Workq, for simplicity, does not have any separates "tubes" however. Just a job pinned by its name.