we are so stuck with this push request load balancing its crazy, if we just switch to pull instead of push things get much smoother, and resources get better utilized

you cant reliably guess if the instance where you will push your request actually has capacity to handle it, even using ML to guess it will still have thrashing properties

but if you just let instances pull work, things work out for themselves

sadly, the whole industry is stuck on http push

i was playing with it few years ago making a simple queue and wrote my ideas here: https://punkjazz.org/jack/we-got-it-all-wrong.txt and some basic benchmarks https://punkjazz.org/jack/we-got-it-all-wrong-2.txt

in the same time, there is so much tooling for http, and its so natural to use it, that it is actually hard to switch to another transport layer, so now the best we have is some naive bayesian classifiers in the LB and some exponential backoff

here is the difference i had in timings using synchronous io queue vs http for 2 endpoints, one fast and one slow (endpoints had the same code, just transport was different)

    2019/02/19 22:26:27 synchronous QUEUE ... 20000 messages, took: 29.95s, speed: 667.74 per second
    -   0+10=9711  48.55% ******************************
    -  10+ 5= 276  49.94% 
    -  15+ 8=  20  50.03% 
    -  23+12=  16  50.11% 
    -  35+18=3398  67.11% **********
    -  53+27=6549  99.85% ********************
    -  80+40=  30 100.00% 
    - 120+60=   0 100.00%
    - 180+90=   0 100.00%
    - 270+ 0=   0 100.00%


    2019/02/19 22:26:56 http ... 20000 messages, took: 29.55s, speed: 676.91 per second
    -   0+10=3274  16.37% ********************
    -  10+ 5=1523  23.98% *********
    -  15+ 8=4439  46.18% ****************************
    -  23+12=4756  69.96% ******************************
    -  35+18=3318  86.55% ********************
    -  53+27=2037  96.73% ************
    -  80+40= 581  99.64% ***
    - 120+60=  70  99.99% 
    - 180+90=   2 100.00% 
    - 270+ 0=   0 100.00%

you can see how the fast endpoint is always fast with the queue transport, but with classic push load balancing it spills latency a lot, because the instance is sometime busy servicing the slow request

PS: this post is absolutely amazing! and the animations are brilliant! thanks a lot for making it