* choose a language that supports async operations so that you aren't waiting for external webrequests or database calls to get back to local code.
A synchronous language can turn your big, beefy machine into a single concurrent transaction language and _that_, beyond any other pain, can really, really, really reduce your total throughput.
Pick languages that can do other things while they're waiting for IO.
At $WORK, we run a lot of 1000s of php servers with 10-20 processes each using nginx. External calls we don’t need the result of immediately get kicked out to an external queue, but we haven’t seen any issues with DB calls blocking the whole server.
Long or slow DB connections are an issue for individual request time and DB load though (the biggest DBs we run handle 1000-2000 concurrent connections).
Right now in a microservice model often you have to remind developers that it just isn't ok to have a ~1ms response in a basic case and a 1 second to infinitely blocking case during errors or slow remote services.
People are saying that python and ruby can't do things in parallel in an easy way.
While technically true, it is also pretty much irrelevant, since we have multithreading.
Multithreading just increases your technical max concurrent request up to the size of your multithreading maximum.
Async lets the computer do _anything else_ while it's waiting for the request.
If you're handling thousands of concurrent requests, you need thousands of threads available ... or, 10s or hundreds of threads with async available (your mileage may vary).
If you prefer some other paradigm then Concurrent Ruby[1] has a swiss army knife of them including event loops and Go style SCP.
I think things will improve in ruby 3 but even without ractors it's still possible to do either async IO in MRI or use JRuby for fully parallel ruby.
Ruby's had aysnc IO support for at at least 10 years I think.
Rails itself has been using Concurrent Ruby since at least Rails 5 but I think it had its own concurrency patterns even before that. e.g. ActiveRecord database drivers.
Jesse Storimer's Working with Ruby Threads book from 2013 is still a really good resource for concurrency in ruby.
It regularly took down the multithreaded python instance.
The solution we went with was to put all the different web requests into 2 flask servers and then take the output of all those different web requests and make one big request tot the django instance that had all the business logic.
If memory serves, we only had 2 flask instances handling all the traffic that the old array of django servers.
Async pays dividends.
edit: wording
Python, until recently, didn't.
Even then, it's a bit misleading; make it so concurrency, in general, is easy and efficient. Async is immaterial if you have green threads or similar (a la Go or Erlang). Point is, you shouldn't risk a model that ends up unnecessarily blocking/synchronizing execution.
Do you have any examples of this being used with Rails? I'm having a bit of trouble finding examples of both Socketry and Rails being used concurrently; and, I will admit, my Rails knowledge is a bit .... intermediate or less.
https://docs.python.org/release/1.5.2p2/lib/module-select.ht...
https://yehudakatz.com/2010/08/14/threads-in-ruby-enough-alr...
https://github.com/tornadoweb/tornado/tree/branch1.2/tornado
That's the earliest I could find off the top of my head - But the other comment responding to you found an even earlier one where they mention the "select" feature that was added sometime around 1998 (22 years ago). The twisted async networking library seems to have been released/started at least 13-14 years ago as well.
I think we all (myself included) kinda forget or underestimate how old Python really is.
If something is CPU bound it is likely to be memory latency unless it is very optimized and running on multiple cores.
If something is not CPU bound neither memory latency or memory bandwidth will be the current bottleneck because both show up as CPU time.
Prior to the new async stuff, though, Python was a horrible language if you had internal microservices bouncing requests off each other - the edge layer would wait a very long time for all those secondary requests to finish before it could respond.
* Don't pick languages with no concurrency support.
If you know more, the thing to do is to share some of what you know, so the rest of us can learn. If you don't have time or don't want to do that, that's fine of course, but then please don't post.
In case you're interested, a more in-depth explanation of this principle is https://news.ycombinator.com/item?id=25130956, from yesterday.
Edit: it looks like you've unfortunately been posting a lot of unsubstantive/flamebait comments. We ban that sort of account because we're trying for curious conversation here. If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html and sticking to the rules when posting, we'd be grateful.