They also had a famous fight with heroku
https://genius.com/James-somers-herokus-ugly-secret-annotate...
They produced an intelligent, thorough, and fair exposè of poor engineering practices by Heroku which were otherwise not documented (or worse, misleadingly documented), and which also benefited Heroku financially, since users would have to purchase many more dynos to get the expected performance - up to 50 times more if I recall correctly.
Genius didn't have to share any of their analysis. They could have simply shown it to Heroku and probably received a bunch of "hush" benefits to forget about the whole thing. The fact they bothered to polish it and open source it allowed others to benefit from their work and meant that even 8 years' later, someone like me can better plan hosting requirements and avoid falling into the trap they did.
Yes, Heroku's load balancing works the same way to this day, resulting in much higher costs to medium-large apps as well as huge degradations to end user experience as some users will randomly be routed to dynos with long queues despite others being completely idle! This is a huge waste of energy and bad for the environment too.
As a heavy Heroku user, I'm incredibly grateful for Genius's work on this issue!
Or it’s simply that Heroku didn’t provide customers great visibility into a the issue at hand, that the best-in-class at the time 3rd party add-on that gave that visibility was misleading, and that architectural changes required to support a broader set of languages in a more distributed fashion introduced a change in behaviour that was not documented. The engineering best practice continues to be the same, which is why it is unchanged 8 years later: push long-running requests into an async flow. It’s both more scalable and more durable.
Consider Genius's checkout analogy: busy markets are efficient because customers sort themselves into the shortest queues. If customers were randomly allocated queues, some customers will wait in lengthy queues while others queues remain empty!
Critically, the same logic holds whether customers have lots of items (long-running requests), or very few items - one is slightly worse, but customers are still waiting unnecessarily while other queues are free to be used (but aren't).
So it's a problem even for applications with no long-running requests. Suppose a user's request falls in a queue with 10 users being served first, while other queues are idle, also assume requests are reasonably fast - 3000ms - then you still have to wait 33 seconds (!!) for the "3000ms" request to be served (!!) - I think that will error because heroku times out requests at 30000ms. This is a disaster for UX.
Someone made a simulation to show how incredibly wasteful it is: https://gist.github.com/toddwschneider/4947326
Note that while it would be great for it to be improved, I in no way demand/expect that, I just don't think it's fair that Genius aren't given credit for their analysis - analysis that Heroku ought to have provided. It shouldn't have been Genius or other customers to expose that Heroku had misguided them, even if it happened accidentally and without intent.
The original “intelligent” routing was possible because the router effectively had a single shared state to track all these things which was also a single point of failure. And because the state of the art of web servers for Ruby back when it was implemented was Mongrel and it was single threaded.
Add in support for nodejs (and other multi-threaded languages to soon follow) and suddenly each dyno can process multiple request concurrently and the request accounting isn’t as valuable. Make the router more fault tolerant and the overhead in trying to manage that accounting at scale becomes almost impossible without adding significant overhead to every customer request and degrading performance for everyone.
I bristle when people seem to think there was some nefarious revenue incentive from Heroku here. They were well reasoned platform changes to improve availability and increase performance for more efficient languages. In the process the marketing got disconnected from the implementation.
A browser extension or something would liberate people from these walled social media gardens of interacting. Just not attractive and a hard sell.
I remember seeing a cringy interview of the founders. Candidly, I'm not surprised they couldn't convert their idea and make it happen with something bigger. Who knows though they did raise the cash.
I still think there is room here though.
There have been many other initiatives, some listed here [1]; Wikalong was interesting but riddled with spam. Maybe a personal system (not shared) would help prevent content abuse? But then it would probably offer too little value to really take off.
See "Notes for an Annotation SDK" from a couple weeks ago <https://blog.jonudell.net/2021/09/03/notes-for-an-annotation...>