What is that breaking point where developers should switch to more scalable models? From the article:
> There is a threshold beyond which the use of synchronous APIs just doesn’t scale
What is a good rule of thumb these days?
What is that breaking point where developers should switch to more scalable models? From the article:
> There is a threshold beyond which the use of synchronous APIs just doesn’t scale
What is a good rule of thumb these days?
This “mostly idle” assumption is gonna break the moment you hit a load spike and wake up most of those “idle” threads.
The success of async is often (mis)attributed to eliminating unnecessary resources, but in practice it's actually because async model implicitly enforces an invisible job queue, where usually only a single process/thread per core is heavy lifting the actual work, and async I/O will feed processing pressure back to the load source via network (e.g. TCP ACK).
Besides that, it’s pretty hard to decide you want to go async later on. In languages where async is not omnipresent, you usually get some form of explicit continuations; async/await sugar if you’re lucky. But you can always accidentally call something not async and hang, or just spend a lot of CPU time with no ability to preempt.
I think Erlang, Go and probably a couple of others basically stand alone in this regard; everything is just async by default. It’s a shame that it requires a decent amount of runtime baggage to accomplish this.
Ruby and Python can only ever have one thread running at a time due to global interpreter lock. Though to keep in mind when I/O happens on a thread it can be parked and another thread can be picked up to continue processing. But in essence one thread is running at a time and these languages built billion dollar businesses.
Just scale horizontally computers are cheap is the message I get and remember to keep it simple.
I'd be very surprised if Robinhood's trade matching engine is done in Python. I'd bet they use an off-the-shelf engine in C++/Java or another language with good multi-threading support.
> computers are cheap is the message I get and remember to keep it simple.
Fully agree! Getting to product-market fit is probably easier with python and ruby.
But, it’s only efficient in convenient, shared-nothing type architectures. If you are dealing with longer lived, more stateful stuff (like a game server perhaps) it’s not going to be so easy.
Of course, it’s easy for fancy VC funded stuff to defer optimizations later. If you are working solo you are probably going to want to, counter intuitively maybe, spend more time thinking about how to keep things optimal and simple. There’s probably startups out there worth real money whose entire stack could run on a $20/mo DigitalOcean box if it were better optimized, but actually runs on a $1000/mo Amazon or GCP account. Of course there is a lot to be said about that (obviously, the latter set up should, if done well, be a lot more resilient and prepared to deal with scale) but it does go a long way to demonstrate why someone might care about efficient async; the scale of money you can burn when you are bootstrapping or doing solo hobby projects is a lot different from well-funded startup or established business...
Everything has tradeoffs. I do believe Go wins in context switching, but the GC latency and CPU usage has definitely caused some problems. The Go developers have done an impressive job with GC latency and performance, but it’s hard to be cheaper than free, so sometimes Rust is a better option, for example.
Async NetworkStreams or Sockets in .NET is a well-designed API over traditional BSD sockets’ select() function which is how you scale to tens of thousands of concurrent connections using only as many threads as you have hardware DoP - it’s not that I’m wowed that .NET has modernised it, it’s that Java’s solution is to wallpaper over a gaping hole in their fundamental design instead.
That is why you will find out lots of code bases full of Task.Run() and direct calls to .Result, instead of using async/await operators.
Having worked with both platform since their yearly days, I am not sure which design is actually the best one, considering brown field development.
What I am describing is what a large majority of .NET consulting projects do, because async/await taints the whole call stack up to Main() or event handlers, and almost no one is doing .NET Core projects from scratch.
Just because it is discouraged at conference talks and blog posts, doesn't mean devs aren't doing it on the trenches when dealing with the crossfire of project deadlines and massive codebases.
Oh, I'm sure of that, don't worry.
I'm just fortunate that I've never had to experience that myself.
What's being introduced now is a way to use that API without needing to touch the whole concept of async code. From the developer's perspective blocking calls and threads is pretty nice. It's async that's awkward and hard to use. Making the existing threads facility scale much better is a very clean approach: virtual threads introduces "virtually" no new concepts at all, in fact, if you aren't working with native code via JNI or Panama then you can act as if virtual and physical threads are the same. You literally just upgrade Java and maybe flip a couple of switches and you can now assign one thread per connection yet handle millions of connections (assuming you don't run out of other resources of course).
Even with Loom's green-threads (not the original green-threads) async IO in Java will be too otherworldly until Java's language designers cease their intrasigence about adding first-class support for `await`.
I'm sorry but I can't take you seriously after you say something like that.
If you start off with a synchronous API and thousands of threads, migrating to something else later is very expensive.
I started writing some fancy asynch code, but on checking with a profiler, the thread overhead was pretty negligable, so I left it alone.