Queues invert control flow but require flow control

The Spintronics mechanical circuits game is sort of like conveyor belts.

Electrons are not individually identified like things on a conveyor belt.

Electrons in conductors, semiconductors, and superconductors do behave like fluids.

Turing tapes; https://hackaday.com/2016/08/18/the-turing-tapes/

Theory of computation > Models of computation: https://en.wikipedia.org/wiki/Theory_of_computation

Like Lucy in the chocolate factory.

Playing Factorio has indeed given me a much better understanding of how congestion works in complex systems.

It also assumes the producer and consumer are monolith-bound data streams. Almost all AMQP back-ends include traffic message-pattern smart-message-routing to pre-sort the data as kind of an implicit sharding operation. This pattern is also particularly performant using static route definitions.

Unless one is using Elixir/Erlang channels, than eventually one must glue on the equivalent queue structure to handle concurrency spikes. Note, this is less to do bandwidth peaks, but rather the OS ISO/OSI limitations will become the bottleneck for concurrency.

If your data traffic is truly separable, than it is trivial load-balancing across multiple pattern-matching routing consumers. Thus, you get pre-sharded localized data, and simply avoid the stores state-syncing cost on back-channel links.

If one has over 40k users, than they should already have re-discovered this phenomena. Simply repeat the mantra "concurrent consumers should never share state". =3

You make a good point: if we’re always able to scale out the consumer, then we never need flow control to slow down the producer.

I’ve worked with on prem and now AWS. Even being on AWS I think the first part of the article is great. Nice visual and description. When on prem, scaling was an issue during event storms. But by then we at least had enough to know what the problem was.

Scaling the number of messages is no longer an issue for us using SQS but size is. Basically there are still constraints. We then need to pass references. “Details created etc find it here”. On prem we could dump around 25 MB into message without issue and could go to 50 MB but it wasn’t safe.

I get your point on the queue and pub/sub not being the same. Just to note we do a lot of hybrid and expect others do to. Publish to a topic and bridge to queues. Fan out is easy and gives choice. I don’t know google’s version, with TIBCO EMS this was easy to manage and was clear to everyone. If you wanted to listen to everything on prod.alerts.* you could or if you want to process prod.alerts.devices.headend you could just queue it and process all.

We use queues like storage for long outages so that senders don’t have to change anything. Not a great use but people were sure happy to know we could help by holding all the events while they could deal with their mess. Never got close to any limit when doing this on prem.

Never used it but isn’t the idea of Kafka to hold all events like a database? Love the idea. Seems so lazy and useful at the same time. Now that I write this I can see the danger too. You become a transaction system of record. Ugh. That’s someone else’s problem ;).

> Flow control is unnecessary > compensate (ideally by autoscaling)

Isn't that flow control of a different form? It's a new form enabled by the cloud moreso than any previous technology, but it really does seem like a version of "egress based flow control" to me; except, we're widening the output rather than narrowing it.

Queue failures are just a destructive form of back pressure.