We Don’t Run Cron Jobs (2016)
engblog.nextdoor.com
engblog.nextdoor.com
https://github.com/finix-payments/jobs
It compiles to machine code, so deploying the binary is easy. That said, I’d like to add some tooling to simplify deploying and configuring Postgres and PostgREST.
Edit to add: oh yeah, I just read another comment that reminded me that it also ate up lots of disk space as well.
Edit: I should expound. Jenkins seems like it has a lot of clunky moving parts. It all works, and I’d rather use it than anything else, but it’s kind of like IKEA furniture: you use it because you have to, not necessarily because you want to.
It’s also incredibly difficult to automate. I can configure Postgres with a config file or two and easily use Ansible to get the exact same instance every time. Jenkins has to be dragged into Automation Alley kicking and screaming. I partially blame this on the fact that Jenkins has nontrivial amounts of configuration that’s done via GUI. I approach a long-running Jenkins instance with the same fear and dread I approach a Windows box that hasn’t been restarted in six months. I.e. the box is now a snowflake and trying to make it reproducible and automated is going to be a bad time.
I could go on, but as a devops critter, Postgres wins every time.
It's like comparing a missile with an airplane. One gets where it needs to go faster and more efficiently, and the other one transports people.
Sure it works but from an admin perspective it’s horrid.
Only you don't. I'm not sure why you seem afraid of Postgres, but millions of people use it, across tons of companies, and even for personal projects, and it's trivial to setup and run. Oh, and those lots of plugins and stuff you mention? You don't have to use them if you don't need them. They don't even enter the picture at all.
Because I've managed database applications before?
> and it's trivial to setup and run
Jenkins is more trivial. It's one process. It doesn't depend on an external high-availability networked data service. Backup is 'cp -r SRC DEST', or a plugin if you're fancy. And, again, Postgres does not replace Jenkins, it's just the storage and querying. It adds a lot of complexity and service availability points of failure that Jenkins does not have.
Copying the jenkins directory can be done but restoring it on another fresh install will not work. There are many files that needs to be deleted manually until the instance can start without error.
pg_dump on the same data volume takes about 14 minutes to dump, compress and move to another node.
The filesystem is a shitty database. Thought we’d all learned that by now.
How dare you :) "The" filesystem is a great database.. for certain applications. Big, binary blobs of video, store, and even index, particularly well!
I think what the collective "we" haven't learned is to avoid trying to think about scale intuitively (rather than "doing the math") and to avoid extrapolating from the trivial scenario.
It's why "latency numbers every programmer should know" is still a thing.
Heavily locked stuff, lots of small things, huge number of locatable data entries, not so much. Which is Jenkins.
As for scale, Incidentally I worked on a very old filesystem based store back when we had spindles to contend with. We had four racks of Sun disk array trays. The only way it performed well was keeping it to 2Gb spindles. I’m well aware of scale issues on file systems, perhaps moreso than the Jenkins developers. We had to scale that to 4000 concurrent users.
My comment was partly in jest, and mostly hoping to spur conversation, not as a true disagreement or criticism with your comment.
Still, saying "It's OK" for those narrow use cases may be too dismissive, even if "great" is an exaggeration. There are plenty of examples where DBMSes (especially relational ones) have fared poorly in comparison.
> Outside of that it needs something that has some intelligence
I fear I'm missing your point here. Certainly "the filesystem" as in the Unix syscall interface to a hiearchical arrangement of files, lacks intelligence, but that doesn't mean the specific underlying implementation must.
We've even come a long way from every being the Berekeley Fast Filesystem. Besides the many choices of underlying filesystems (including CoW ones like ZFS and BtrFS)
> and ability to read optimise it.
I assume by read optimization you don't just mean something like the buffer cache, but a user-specified index?
> Heavily locked stuff, lots of small things, huge number of locatable data entries, not so much. Which is Jenkins.
Does Jenkins use external locking? It would be odd in light of some of the comments elsewhere in the thread touting its advantage of being a single process. Of course, even if it's using only locking internal to itself, there's a good argument that its authors needlessly re-invented a DBMS (which we've seen happen when other niche-use databases get used for broader purposes).
I'm not sure a large number of tiny files is inherently problematic for a filesystem, merely problematic for existing implementations, and some are better at it than others. What about something like libferris (assuming perfectly spherical cows and ignoring the performance implications of FUSE for a moment), which can back a filesystem with an arbitrary database?
IOW, is Jenkins-using-the-filesystem an Ops optimization/tuning problem, or is it a more fundamental problem that can only be addressed with modifying its code?
> The only way it performed well was keeping it to 2Gb spindles.
I worked with the aforemention hardware extensively, early in my career, but at a company that wrote a data warehousing (aka OLAP) RDBMS.
I'm reasonably confident in saying that your performance observations have almost nothing to do with the filesystem itself and everything to do with I/O performance in general. Large numbers of smaller spindles were absolutely required good database performance and scalability.
> I’m well aware of scale issues on file systems
I didn't mean to suggest you didn't, rather the opposite, as "the collective we" was a euphemism meant to imply "everyone but us".
Anyway, my overall point is that you and I may be acutely aware of the real, practical problems with scaling filesystems, but we're rare. Since there's nothing fundamental/theoretical that makes the filesystem an obviously poor choice at modest scale (the definition of which increases as computing power increases), the lesson does not get learned by everyone.
Instead, because truly large scale becomes rarer and rarer as computers become more powerful (CPU more than I/O, of course, but then.. SSDs), every time the lesson is re-learned by an individual/company, they think it's a new, or at least unique problem, and we end up with a re-invention of Portable Batch System (a fairer characterization than re-invention of cron, IMO).
https://github.com/geerlingguy/ansible-role-jenkins
https://github.com/geerlingguy/ansible-role-postgresql
Two ansible roles by the same author, supporting both Centos and Ubuntu. Not hugely different in complexity IMO. Installing Postgres on FreeBSD, though, is little more than
pkg install postgresql10-server
sysrc postgresql_enable=“YES”
service postgresql initdb
service postgresql startHow does Jenkins HA compare?
I've only ever used it in the internal/build scenario, never in production.
Docker containers run using a task scheduler are perfect for this.
* Uwe Ohse's uschedule, http://jdebp.uk./Softwares/uschedule/ https://ohse.de/uwe/uschedule.html
* Bruce Guenter's bcron, http://untroubled.org./bcron/
* GNU mcron, https://news.ycombinator.com/item?id=17002098
* Thibault Godouet's fcron, http://fcron.free.fr/
* Matt Dillon's dcron, http://www.jimpryor.net/linux/dcron.html
Other toolsets include Paul Jarc's runwhen (http://code.dogmap.org/runwhen/) which is designed for first-class services individually scheduling themselves.
For a distributed cron, you need a more sophisticated, transactional system that can deal with node failures.
There are many cron replacements, but they generally don't tackle the main problems with the original cron, such as the inability to retry, queue, prevent overlapping jobs, enforce timeouts, report errors and metrics in a machine-readable way (spewing out emails is not a good solution), etc.
Those seem to be rarer, since the legitimate use case for a distributed system (what used to be called "grid computing") were rare.
Nowadays, I assume everyone just uses Yarn on Hadoop, when they think scaling "up" ends at a mid-range 2U server. I don't know how good its actual time-of-day/calendar based scheduling is, though.
For example: Want to run a backup every night? With cron, you run into several issues: A backup job could fail; how do you recover? A backup job could run (due to sudden latencies) for an unexpectedly long time, overlapping with the next scheduled time; how do you prevent "dogpiling"? How do you record the complete log about when each job ran, what it output, and whether it was successful or not? And so on. Or for that matter: What if the box that is supposed to schedule this job goes down?
These are fundamental systems operations tasks that you want Unix to solve. Unfortunately, cron leaves all the actual hard challenges unsolved; it's fundamentally just a forker. Cron, then, isn't really useful for much except extremely mundane node-local tasks such as cleaning up $TMP. I can think of relatively few tasks in a modern environment that can use cron without running into its deficiencies.
This means that, for example, backups tend to be handled by an overlapping system that actually has these things built in. This is a shame, because the Unix philosophy wisely encourages us to separate out concerns into complementary tools that fit together. Instead of a rich, modular, failure-tolerant system that only knows how to execute jobs, not what the jobs are, you get various monolithic tools that build all of the logic into themselves.
Not "everyone just uses Yarn" at all. For the projects I work on, we use distributed cronjobs on Kubernetes, which solve pretty much all of the problems with cron. For many people, both Hadoop and Kubernetes are overkill, though, yet they would benefit from a resilient batch scheduler.
I'm a bit confused. It still sounds like you're describing a batch scheduling system and that cron isn't one (because it only implements a narrow function of such a system).
What features does PBS (not cron) lack that any of the re-inventions do have?
> Unfortunately, cron leaves all the actual hard challenges unsolved
> the Unix philosophy wisely encourages us to separate out concerns into complementary tools that fit together.
I'm also having trouble reconciling these two positions. Cron does one thing, which is forking on a schedule. To leave something like "dogpiling" for another utility (e.g. dotlockfile) to solve seems consistent with the Unix philosophy.
> Not "everyone just uses Yarn" at all. For the projects I work on, we use distributed cronjobs on Kubernetes
That doesn't quite refute my point, as you can feel free to consider "Yarn" as merely a metaphor for the currently most-popular inbuilt scheduler of a currently popular distributed computing platform.
Talk about reinventing the wheel!
Jenkins is leaky, unstable, unfriendly to the filesystem, buggy and poorly documented and tested.
If you’re going to do anything like this it’s actually worth paying for team city or something else.
In a gig last year, I needed to improve the build turnaround times on a Jenkins system. After learning everything I could about Jenkins, I realized the correct answer was to delete it and rewrite my own version that ran on the local system, which was way, way faster and much easier to debug and maintain.
Not having to commit/upload your code to a build server and then wait to get an executable/package back is an enormous time-saver just in that overhead alone, but even the build itself was faster, even though it was written entirely in bashscript.
Go figure.
"Maybe next year we'll have time to look into your plugin."
In any system I manage the cost of the plugin simply isn't an issue - if it's worth the money, it's easy to justify paying. But if it's free, and it becomes critical to the workflow, and in a year's time the author has abandoned it, then the "cost" is that we assume the responsibility for maintaining it forever, or we retool the workflow, or in some other way, it's very expensive. So paradoxically paid-for plugins are a much easier sell and requests for free stuff are shot down straight away.
We ended up writing 114 line typescript job scheduler that uses Mongodb as a store. Mongodb provides the atomicity of scheduling (find and modify). Beyond that our UI is Robo 3T. The job collection has 4 simple states.
Any process can write a job document with the earliest time the document can run. It is also scaled with the rest of our application: more instances, more jobs that can run simultaneously.
But... I can see in shops that don't have this complexity yet, Jenkins might be just fine.
Edit to add: We decided against using another data source like a Queue which is something we would have used a long time ago, but we're already at an infrastructure complexity point where it would not be worth it - we already have Elasticsearch, MongoDB and Mysql so adding a AWS queue or rabbit would be out of the question at this point unless it provided a massive functionality we would need for something else.
It sounds like you have a very simple collection of jobs running that lack run-time complexity like remote hosts being unavailable or preventing a service from being overloaded with requests after it comes online because you have an ever-growing list of tasks waiting.
The actual complexities of DAG-based workflow management tools are considerably more complex than can be expressed in 114 lines of any language, and I hope any programmer I work with would spare me and my peers the misery of trying to roll and maintain our own when plenty of open source and actively maintained projects fit the bill that could benefit from our contributions.
I don't know why you would mention this because you don't have the requirements I have. If I had solved all scheduling needs for all people in 114 lines of code I would have written so.
Nothing you go on to detail after that opening sentence has anything to do with Jenkins itself but rather your company and it's organization.
I think this is a case of everything looking like a nail to someone with a hammer. Just because you need a solution for scheduling and jenkins does scheduling doesn't make it the right tool for the job.
Hopefully everyone recommending it is at least running separate instances for production environments? Some (thankfully) former colleagues of mine had our build server running production jobs...
Builds are just another job, usually triggered by a web hook after a git commit or periodic polling.
At my last job, we took cron jobs that were spread across 27 servers that we're really well monitored and moved them to Jenkins. The server SSH'd into the boxes (where necessary) ran the jobs, alerted us if there was a problem in Slack and we could use the tracked logs to find out exactly what happened. Really helpful.
Plus the cron scheduler has an alternative to picking a specific time to run by deferring to Jenkins based on other things that are running and the time it historically takes to run the job.
So instead of a bunch of jobs running at 3am you can set a job to run at [1-5]H and it will run sometime between 1-5am as best determined by Jenkins.
You can also trigger other jobs following the success (or failure) of another. For example, we used to have a daily digest that ran at the same time everyday and backed up our workers for a couple of hours. Instead, we created worker servers that only listened to that specific queue and scheduled a Jenkins job to start & update however many we needed, then after it was successful to trigger the digest jobs. Later in the evening, it was scheduled to scale down.
Just having all the jobs in one place with job specific log rotation rules and success tracking was extremely helpful.
It wasn't pretty...but it worked really, really well.
Everything is fully Dockerized, and Jenkins is set to just run the docker image with a specified entry point.
Different requirements, different solutions.
No doubt this works if it's an established procedure, but if I was approaching a system I wasn't familiar with I would never do (3) because the environment can differ wildly between crond and a login shell. It is safer to edit the cron schedule and duplicate the entry with a time set for few minutes in the future. (And clean that up afterwards).
Just use Jenkins, until you outgrow Jenkins. Then use a distributed scheduler.
We recently had to implement a prototype in house, not that easy as I thought it would be, but we could not find a good solution, e.g. handle 100k jobs, long term (never expiring), short term (depends on an entity's live cycle), high available.
Thanks.
If Cron doesn't work, get an open source or commercial solution. And who cares what tech the scheduler is written in? Scheduler's job is to provide run your programs and provide API and a nice GUI if you desire.
[1]: https://engblog.nextdoor.com/nextdoor-taskworker-simple-effi...
The single biggest advantage this has gotten us is centralized logging. I can check on the console output of any cronjob just by going to Jenkins and clicking on the job.
Moving to Jenkins to cron wasn't my idea, but the implementation is mine. I've built a few base Docker images as bases. One is just the standard Python 3.6 docker image. Another is the CentOS image equipped with Python 3.6 and Oracle RPMs for jobs that need database access. Another is the aforementioned image plus a number of Perl dependencies for jobs that need to call into our legacy Perl scripts.
For many scripts, I can use identical Dockerfiles. I just copy the directory containing the script, requirements.txt, Dockerfile, and Jenkinsfile, then I change out the script, edit the Jenkinsfile to reference the new script's name, and make any needed changes to the requirements.txt.
That's great, what does that have to do with cron?
As a result what I read was:
"We don't understand what cron does, nor do we understand how job scheduling is supposed to work, and we don't understand how to write 'service' based applications, so somebody said 'Just use cron' and we did some stuff and it didn't work how we liked, and we still haven't figured out really what is going on with schedulers so we wrote our own thing which works for us but we don't have any idea why something as broken as cron has persisted as the way to do something for longer than any of us has been alive."
I'm not sure that is the message they wanted to send. So lets look at their problems and their solution for a minute and figure out what is really going on here.
First problem was 'scalability'. Which is described in the blog post as "cron jobs pushed the machine to its limit" and their scalability solution was to write a program that put a layer between the starting of the jobs and the jobs themselves (sends messages to SQS) and they used a new scheduler (AP scheduler to implement the core scheduler).
So what was the real win here? (since they have recreated cron :-)) The win is that instead of forking and execing as cron does allowing things like standard in and what not to be connected to the process, their version of cron sends a message to another system to actually start jobs. Guess what, if they wrote a bit of python code that all it did was send a message to SQS and exit that would run pretty simply. If they did it in C or C++ so they weren't loading an entire interpreter and its run time everytime it would be lighting fast and add no load to the "cron server", this is basically being unaware of how cron works so not knowing what would be the best way to use it.
Their second beef was that cron tabs are hard to edit reliably. Their solution was to write a giant CGI script in a web server that would read in the data structure used by their scheduler for jobs, show it as a web page, let people make changes to it, and then re-write the updated data structure to the scheduler. Guess what, the cron tab is just the data structure for cron in a text form so you can edit it by hand if necessary. Or you can use crontab -e which does syntax checking, or you could even write a giant CGI script that would read in the cron file, display it nicely on a web page, and then re-write as syntactically correct cron tab when it was done.
Problem three was that their jobs were complex and failed often. This forced their poor opsen to log in, cut and paste a complex command line and restart the job. The real problem there is jobs are failing which is going to require someone to figure out why they failed. If you don't give a crap about why they failed the standard idiom is a program that forks the child to do the thing you want done and if you catch a signal from it that it has died you fork it again[1]. But really what is important here is that you have a configuration file under source code control that contains the default parameters for the jobs you are running so that starting them is just typing in the jobname if you need to restart or maybe overriding a parameter like a disk that is too full if that is why it failed. Again, nothing to do with cron and everything to do with writing code that runs on servers.
And finally their is no telemetry, no way to tell what is going on. Except that UNIX and Linux have like a zillion ways to get telemetry out, the original one is syslog, where jobs can send messages that get collected over the network even, of what they are up to, how they are feeling and what, if anything, is going wrong. There are even different levels like INFO, or FATAL which tell you which ones are important. Another tried and true technique is to dump a death rattle into /tmp for later collection by a mortician process (also scheduled by cron).
At the end of the day, I can't see how cron had anything to do with their problems, their inability to understand the problem they were trying to solve in a general way which would have allowed them to see many solutions, both with cron or with other tools that solve similar problems, would have saved them from re-inventing the wheel yet again, giving them the opportunity to experience the bugs that those other systems have fixed given their millions (if not billions) of hours of collective run time.
[1] Yes this is lame and you get jobs that just loop forever restarting again and again which is why the real problem is the failing not the forking.
EDIT: Oh by the way, Ruby folks struggling with cron might appreciate this: https://github.com/pjungwir/cron2english/
I recognize that it is a sore spot for me when people re-invent the wheel when it seems clear they didn't have to and even clearer that the energy spent re-inventing the wheel would have been better (in terms of using the wheel) spent learning why the wheel is the way it is.
It is sort of the Chesterson's Fence of computer science. Don't tell me its wrong, tell me how it is right for what it does and where it falls short for what you want it to do. Cron gets a bad rap here, as did sendmail for that matter. When taken to extremes you replace working systems like init with systems that are borked like systemd. No doubt cron is on the list of things to be absorbed by the systemd borg at some point. Yes, it makes me grumpy.
This has already happened, right? https://wiki.archlinux.org/index.php/Systemd/Timers
Not a systemd fan, myself.
I understand some of the systemd criticism, but I don't at all understand why you want something else to manage your processes than the process management system (which is basically always the init system)?
Perhaps you don't understand one of the fundamental, philosophical criticisms, which is that it violates the Unix tenet of "do one thing and one thing well".
Timing, and scheduling are difficult and (arguably, of course), deserve their own, separate utility.
Cron isn't a process manager, any more than and interactive shell [1]. It just starts processes (and passes output to a local email client, though that's, understandably, brittle).
I wouldn't object to moving the process-starting functionality out of cron and leaving it with only the ability to tell init to "start job named XYZ", but I do object to just integrating its core competency of scheduling into that init.
[1] Arguably less than, considering modern shells' job control and signal facilities
The re-invention of this one is one of my pet peeves, especially since the vast majority of the (legitimate) complaints about early implementations have been addressed in modern (last.. 10ish years?) implementations.
Syslog is lightweight, flexible, and plain text.
Back when the ELK stack first started gaining popularity, I'd get asked in interviews if I had experience with provisioning "high volume" logging. I was reasonably convinced that none of those people had ever seen (or would ever see) a high enough volume to be remarkable.
In hindsight, it was probably the same as thinking one has "big data" if it doesn't fit RAM on ones laptop (or even, more charitably, a server, even a decently large one).
It's ironic considering that syslog is not standardized. Plug any two syslog implementations and they won't work together.
The two main competing protocols are defined in RFC 3164 and 5424, both named syslog, they are incompatible with each other and have their own peculiarities.
Logstash, the L in ELK, only parses a subset of RFC3164 messages with very specific formatting.
I don't find any irony there, at all, since it's reflective of the history of Unix, and consistent with the OC's "like a zillion ways to get telemetry out". More to the point, it's fairer to say that different syslogd implementations were parallel evolution, rather than re-invention.
Now, it seems you're talking about the network protocol, whereas "syslog" the system call seems not to be too different across platforms, certainly not to the point of incompatibility. I call this out, not for pedantry, but to illustrate that, from the app's perspective, it's all the same.
> Plug any two syslog implementations and they won't work together.
Unless, perhaps, one of them understands (or can understand, with plugins) more than one protocol?
Regardless, the solution requiring the least engineering effort is to make sure one has compatible wheels everywhere, not re-inventing the wheel.
> The two main competing protocols are defined in RFC 3164 and 5424
I'm not sure that it's fair to call them competing when 5424 explicitly obsoletes 3164 (and the earlier one is "informational", while the later one is a "proposed standard").
Of course, that says nothing about the reality of the competing implementations, but if there are only two, and the older one is the most popular (which I believe it is), then perhaps there's no real competition.
> Logstash, the L in ELK, only parses a subset of RFC3164 messages with very specific formatting.
That would be an example of a (partial) syslogd re-invention. I have no idea if that input plugin is worth using (e.g. for performance reasons) rather than using something like a pipe plugin for plain text from a local syslogd. I suspect for the vast majority of use cases, it's totally irrelevant.
The two RFC are incompatible, then every implementation is slightly different in the handling of miliseconds/microsecond/timezone, the presence of the hostname field, structured data support, etc...
I am calling out your first post that syslog is lightweight, flexible and plain text. syslog suffers from serious issues.
Agreed, but you now seem to be implying that this just happens because syslog, rather than misusing the tool.
Earlier, you asserted "Plug any two syslog implementations and they won't work together" which I didn't bother to contradict, because it seemed like mere exaggeration. However, at this point, it seems like an extraordinary claim requiring extraordinary evidence.
I have, of course, routinely, in my professional career, spanning decades, successfully used different implementations with each other, usually because (more than) one implementation is embedded and can't be changed to match what's on the servers.
Exactly which syslogd implementations connected to each other over the network will fail to parse messages and drop them?
> The two RFC are incompatible, then every implementation is slightly different
I think you're reversing cuase and effect, especially given both the relatively late date of the earlier RFC in the timeline of Unix history, as well as its enduring popularity over the later RFC.
> I am calling out your first post that syslog is lightweight, flexible and plain text.
I don't believe you've addressed, in any way, the assertion that it's lightweight.
Your challenge to its flexibility is, AFAICT, poor interoperability of only the network protocol, which you haven't backed up with evidence. Even then, that's a weak refutation of the flexibility of the mechanism overall.
Your challenge to its being plain text is, again, primarily based only on network protocol, and, even then, based on a widely-unimplemented RFC.
> syslog suffers from serious issues.
This may be responding to an implied strawman, that syslog somehow perfect or ideal for all use cases, which, of course, I never said. In fact, being plain-text could be considered a "serious issue" for some telemetry.
I've used attitudes towards cron, specifically, as a personal metric for judging someone's skill and output with systems. What this team did is exactly the wrong thing to do.
I would rather use the tool with over 40 years of development work gone into it than whatever this is.
High resource usage so they moved the actual jobs off-server. Not cron related. Crontab is hard to edit so we'll create an arbitrary data structure that'll be both hard to edit and non-standard. Good job. Our jobs are failing so instead of better error handling we'll just run them again manually. Great. And apparently they've never heard of cron logging to syslog.
Doesn't every crontab in existence have a comment line giving the order of the time columns? I know I always rely on such.
Although the article didn't touch on it, this point reminded me of yesterday's discussion about manpages and command-line options. I think it's still the case today that `cron -e` is the way to edit the crontab whilst `cron -r` (the key right nextdoor in case this part needed stressing!) removes it altogether.
#!/bin/sh
# print a crontab entry for 2 minutes into the future
date -d "+2 minutes" +"%M %H %d %m * /path/to/command"I'm calling it a "speed-dial for your APIs". I used to have a VPS that would run out of memory, so easiest way to resolve this was to restart the server. But even that was annoying, so I created a button that would call the API to restart the box. Now I just have to click the button whenever I want to reboot.
Something similar to CRON, I'm currently building a feature in tsk to schedule the tasks.
why reinvent the wheel and not use cron or systemd? why reboot instead of searching for the offending script/process and fixing it?
> so I created a button that would call the API to restart the box
Been thinking about building a 'button' that does this as well. Will check out TSK to see how well it addresses this. Sound idea for sure.
[1]: https://github.com/ondrejbartas/sidekiq-cron
[2]: https://hirefire.io
It doesn't replace every cron job, but it is distributed, fault tolerant, and only breaks when the hadoop cluster backs up. The product mentioned here seems like a good solution for a team that needs to execute linux crons without much overhead.
I've been evaluating it as it seems to provide fault-tolerance, but IMO the documentation could be much better with more examples (e.g. mixing different triggers, configs,.etc)
Can anyone comment on it?
After every job is successfully finished, it should be noted in your db. Every time a worker starts processing a job, it should check the db to see if this job has already been run.
It’s dead simple to set up and use.
"We’ve been using Nextdoor Scheduler for over 18 months and we are extremely happy with it."
That means that what they built pre-dates Nomad by at least 11 months.
Glad to see a lot of interesting and insightful comments in this thread!
Some contexts here:
0. Like any piece of software, this scheduler system is not perfect for every company -- legacy code, # of engineers, skillsets of existing employees, engineering culture, tech stack, business,...
1. Every engineer in Nextdoor needs to do on-call rotation -- 50+ engineers back in 2014 (100+ now?), when the scheduler was built. It's important to run a system that all 50+ engineers have confidence to debug on a Friday night if things go south. There are some blackbox-type alternatives of cron, which may be great for a small team of backend experts to operate, but may not be a good fit for 50~100 engineers with very diverse skillsets & backend experience. But why every engineer needs to be on call? Well, that may deserve a new thread of discussions :)
2. In 2014, there were ~200 production jobs with very different computing characteristics (450+ now). Jobs are owned by different teams and different people with different expertise. Jobs are frequently (every few weeks?) added, removed, and updated. Outages most likely happen during code deployment. It’s important to run a system that works well during deployment (and rollback), e.g., always run the RIGHT version of code for hundreds of jobs.
3. The core scheduler system was pretty tiny, which could be easily understood by any engineer in the company. I remembered it took me a few days (probably less than a week) to finish most of the code. The hard part was productionization, e.g., carefully migrating 200 production jobs to the new system, logging, monitoring/alerting at job level, deployment & rollback process, oncall-related things (documentation/training)… With this simple system, we could easily enumerate various failure situations, so oncall engineers have confidence how to respond.
4. I’m not with Nextdoor any more. But turns out this scheduler system is still working well now: https://github.com/Nextdoor/ndscheduler/issues/33#issuecomme... I think the ROI is pretty good -- a 3-man-month project (from design to run all 200+ jobs on the new system) => running 4 years and counting & easy oncall situation for 50~100+ engineers.
5. Why not Jenkins or other open source alternatives? We investigated a bunch of alternatives. Jenkins is great. Back in 2014, Nextdoor used Jenkins for CI (not sure if it's still true now). We ruled out Jenkins (and Rundeck or the like) for operational reasons, e.g., challenges to integrate with existing code deployment process, operational complexity for 50+ engineers with different backgrounds/expertise... Open source alternatives? Well, in 2014, we couldn't find a good Python-based project. It's important to limit the number of languages/external technologies in the tech stack.