Hacker News Reader

Comment by Scubabear68 | Hacker News Reader

I can’t believe an org of Slack’s size relied on cron scripts for anything critical. It is the worst possible way to schedule jobs at scale. Serious problems with discoverability, single points of failure, and failure mode options.

Also surprised they didn’t just use an open source scheduler or product. There are a gazillion of them.

View on HN

Why is this so hard to believe? This is proof that simple software can scale very well. Your takeaway, IMO, should be that a lot of solutions are over-engineered relative to the businesses they serve. Not every company handles Slack's scale.

It’s no longer simple when you have platform code to prevent nodes from disappearing or dying on the minute on a kubernetes cluster the size of slack. A triggered pulse event stream would have done the trick to invoke a lambda or call code for every “thing” that needed a beat. Kubernetes comes with a scheduler…

This is my take as well, just because you can get away with something doesn’t necessarily make it desirable to maintain or extend, and I honestly cannot imagine the effort in terms of labor hours that you’d have to go through to develop something like this compared to just plugging in an off-the-shelf scheduler to something slightly more sophisticated like k8s or even just a worker-and-queue system. When you’re talking about platform engineering to solve a problem that a relatively extensible Celery service could do (and have tests and such), I have no idea how the former could be “less work” or cheaper in the long haul.

Hard to believe because I have seen the cron dumpster fire at so many companies. Random cronjobs running on random boxes ends up with mystery jobs whose existence often goes unnoticed as people leave the org and docs are out of date. Random accounts with different cron jobs, with no visibility from one account to another. Box failure leads to mystery jobs failing with no way to figure out what happened. No “advanced” features like retry and retry with exponential back off. No database with job history. No parallelism.

Every place I have worked cron turned into a dumpster fire.

Sure, but those are organisational failures, not technical failures. If you need better technology because the org sucks at planning, the tech isn’t the issue and you are just sweeping issues under the carpet.

Or you could have a canonical cron box.

Or a fleet? Some of the problems up-thread feel organizational rather than tooling.

No matter what they tell you, it’s a people problem.

i will see your rando cron box and raise you Excel VBA macros run on Windows Task Scheduler on a spare machine in a cubicle.

Never forget that the world economy is probably being held together by a handful of these.

A boatload. Couple. Panamax or bigger. Otherwise agreed.

Cron scaling? Seriously?

By definition cron does not scale, it is account by account per VM/machine with no rhyme or reason.

Alternately, if you define cron as a starting/startup point, an expectation when scaling, then simple format for a job queue, the context of the post is understood.

Yeah that’s the point. If managed properly, turns out it can handle a $20B tech company.

Survivorship bias.

Organisations that cobble together cron scripts for critical apps become an "an org of Slack’s size".

Companies that deploy solutions that handle all the problems with discoverability, single points of failure, failure mode options and interplanetary multi species scale when they have 0 coustomers never become "an org of Slack’s size".

> It is the worst possible way to schedule jobs at scale.

The literal evidence proves that this approach is absolutely workable at scale, so “the worst possible way” clearly doesn’t apply.

The interesting question here is “how did they make this work?” The answer to which is immensely valuable to the technical community at large. “What are they building next?”, whilst interesting, is less immediately valuable as they will be building something Slack-scale, and most orgs are not Slack or Slack-scale

> The literal evidence proves that this approach is absolutely workable at scale, so “the worst possible way” clearly doesn’t apply.

The “worst possible” to me implies that it is possible.

Think about trying to solve the same problem by hurling engineers into an erupting vulcano. It is expensive, hurts the morale, causes staff retention issues, but also fundamentally does not solve the task of scheduled task running. I would not describe that as “worst possible” because it lacks the second factor by not being a possible solution.

Can you name some?

I've been looking for software in that space, a thing that runs cronjobs, with ability to kick off adhoc runs from some UI, see what is running, and if possible parameterise the custom runs (i.e., run an ad-hoc report for a different client than usual cron does)

There's a lot to expore here. First, I'd suggest that cron and shell scripts may not be what you want. Cron has a complex format for scheduling and can lack features like sub-minute level scheduling, maintenance windows, and other task cadences. Shell scripts are OK for small things, but often I find I move to a different language if I keep something around long enough. Do the people writing jobs know shell?

For jobs run by data analysts, airflow and python work great. For devops jobs, begrudgingly, Jenkins or GitHub Actions. But there's so many varieties.

GitHub actions.

I almost suffocated from all the yaml typing this, but unfortunately it’s the baseline.

Build the UI or API for the report, so you can ad-hoc from there. Cron uses that interface. You don't custom the cron, you cron the custom.

Rundeck fits this space.

I almost hate to say it, but Jenkins.

That's what I did. I know it was the right solution because I happily keep adding more and more scripts to run. I didn't do that when I was using cron jobs.

Plus, Jenkins has a few nice extensions to the crontab, including setting a timezone and using "H" to spread job execution load.

> There are a gazillion of them.

Such as?

Reply on news.ycombinator.com