Orchestration isn’t going anywhere

A question to answer is, what does orchestration look like if data flows are continuous, updating as quickly as possible? That's where we're headed. Suppose N sources of data, M places where you want that data to be, and T streaming transformations to apply along the way.

In this world, a separate control plane handling time- and job-based polling has some architectural friction with a data-plane that wants to process records the instant they're available.

Disclosure: I have skin in the game / am co-founder of https://www.estuary.dev/

Here orchestration means in the context of an organization doing some work -- e.g., receiving, making, shipping.

So a challenge is to take the available data and make the needed decisions.

Is this work "going" anywhere? Well, yes, apparently it is.

The step, using the data to make decisions, can involve some math and computing. A lot of the relevant math has been known, and known to be relevant, for decades, but for most of that time computing has been a bottleneck. But the computing is getting to be a lot easier.

Some of the relevant math is optimization. If need to consider that the input data is not always precise, then might want to consider the math of statistics, probability, stochastic processes and optimization in that context -- here, still, a lot is known.

Some of the problems, e.g., at Bell Labs for planning network deployments, quickly led to the math that led to the math of the question, also in computer science, of P = NP??? Is this question "going"? Likely yes, but even if not lots more math/computing is.

> The “orchestrator”... must and will live on. The only question is where. And the answer to that question will be one of the most consequential in data infrastructure in the next decade.

After playing with AutoGPT it's clear this is needed with orchestrating agents who recursively spawn new agents for delegating tasks. Anyone building this?