Ask HN: Where can you get advice on scaling microservices/monoliths rapidly? | Hacker News Reader

Ask HN: Where can you get advice on scaling microservices/monoliths rapidly?

I come across a problem which seems fairly fundamental in the "HN-relevant"/theoretical sense:

When I scale, I have a jobs system with a queue.

Problem is each job can take 2 mins, but I might add a new job every minute sometimes during peak.

If my job system goes FIFO, it'll perform terribly for the latest users, who have to wait 10 minutes or longer.

But if I choose randomly or LIFO, some users never get a solution or have to wait something crazy like 6 hours.

Where can I read about optimising this kind of problem or useful approaches? I feel analysis-paralysis and question each change of default strategy, like FIFO to random order within last 10 minutes (most recent)

View on HN

The question has nothing to do with monoliths or microservices.

The most obvious way to scale in the scenario you describe: process the queue asynchronously, in parallel. By analogy, if a grocery store has a long queue for checkout they put more checkers to work. So you would add listeners to the queue until the jobs got processed as fast as they come in.

If you can't process the queue in parallel, you have to speed up the job processing so the time to finish a job gets lower than the mean time between new jobs entering the queue. Maybe adding hardware capacity (vertical scaling) will work, no way for me to tell from your description. Maybe the jobs can get broken down into subtasks that you can run in parallel.

Fiddling the ordering in the queue doesn't seem like the answer. You need to remove the obstacle that causes jobs to sit in the queue, which appears to come from the long job processing time, not the queueing algorithm.

Fair enough.

I suppose in the current situation, we have too much demand and scaling isn't so much a business priority relative to other ongoing events, so it's kind of the startupy view of tech I'm looking for.

I think you are right that the queue is a case example rather than the true problem, so to say.

The parent reply hit the nail on the head with this:

>Maybe the jobs can get broken down into subtasks that you can run in parallel.

Unless you've explicitly called out that your jobs are CPU heavy, I'm willing to guess that there's a lot of IO/network calls meaning your jobs are taking 6 mins.

If that's the case, and those network calls are to a third party, then you're actually sitting on untapped scalability from their ability to handle calls by not splitting your jobs up into tasks at the point at which they wait for network calls to finish.

It feels like this but every time we remove such an issue (and you're right, it's usually I/O or OCR/ML inference), we just get another.

Whether it's the cloud blocking new GPU allowance increases for a week (I think this is true for most small projects right now), or the fact we know certain traffic will be for at most 2 weeks and so isn't that useful, we just struggle with the strategic view of managing demand. Just feels like plasters after plaster and hoping we won't realise on Monday that half of users had extremely slow experiences due to a new bottleneck.

It's almost like doing fast for 80% of users and never working for 20% is much worse(for UX and future business), than slow for all 50% if you know what I mean because of the importance of reliability... but there are basically few discussions that are this high level... that you can talk in understandable terms to the rest of the C-suite with. Sorry to sort of rant... maybe a first dev ops hire is best.

Nobody said this was going to be easy.

Read the DevOps classics: The Phoenix Project, the DevOps Handbook, and the Google SRE book.

Reply on news.ycombinator.com