My thoughts exactly, this is a perfect use case for Jenkins.
Talk about reinventing the wheel!
Talk about reinventing the wheel!
Jenkins is leaky, unstable, unfriendly to the filesystem, buggy and poorly documented and tested.
If you’re going to do anything like this it’s actually worth paying for team city or something else.
In a gig last year, I needed to improve the build turnaround times on a Jenkins system. After learning everything I could about Jenkins, I realized the correct answer was to delete it and rewrite my own version that ran on the local system, which was way, way faster and much easier to debug and maintain.
Not having to commit/upload your code to a build server and then wait to get an executable/package back is an enormous time-saver just in that overhead alone, but even the build itself was faster, even though it was written entirely in bashscript.
Go figure.
"Maybe next year we'll have time to look into your plugin."
In any system I manage the cost of the plugin simply isn't an issue - if it's worth the money, it's easy to justify paying. But if it's free, and it becomes critical to the workflow, and in a year's time the author has abandoned it, then the "cost" is that we assume the responsibility for maintaining it forever, or we retool the workflow, or in some other way, it's very expensive. So paradoxically paid-for plugins are a much easier sell and requests for free stuff are shot down straight away.
We ended up writing 114 line typescript job scheduler that uses Mongodb as a store. Mongodb provides the atomicity of scheduling (find and modify). Beyond that our UI is Robo 3T. The job collection has 4 simple states.
Any process can write a job document with the earliest time the document can run. It is also scaled with the rest of our application: more instances, more jobs that can run simultaneously.
But... I can see in shops that don't have this complexity yet, Jenkins might be just fine.
Edit to add: We decided against using another data source like a Queue which is something we would have used a long time ago, but we're already at an infrastructure complexity point where it would not be worth it - we already have Elasticsearch, MongoDB and Mysql so adding a AWS queue or rabbit would be out of the question at this point unless it provided a massive functionality we would need for something else.
It sounds like you have a very simple collection of jobs running that lack run-time complexity like remote hosts being unavailable or preventing a service from being overloaded with requests after it comes online because you have an ever-growing list of tasks waiting.
The actual complexities of DAG-based workflow management tools are considerably more complex than can be expressed in 114 lines of any language, and I hope any programmer I work with would spare me and my peers the misery of trying to roll and maintain our own when plenty of open source and actively maintained projects fit the bill that could benefit from our contributions.
I don't know why you would mention this because you don't have the requirements I have. If I had solved all scheduling needs for all people in 114 lines of code I would have written so.
Nothing you go on to detail after that opening sentence has anything to do with Jenkins itself but rather your company and it's organization.