Systems smart enough to know when they're not smart enough (2017)

I'd picture it like the lower-level system having a try{} block (with no corresponding catch clauses), the postdominant block of which would be where you'd resume from if there'd an error within that block that gets caught and resolved. The mechanism knows that the policy needs to have a clean "cut point" (in AOP terms) to subsume it by; but the mechanism doesn't need to know what the policy is going to do at that cut point.

> Moreover, the edge-cases are more or less the sum of what the normal approximators can't handle, "the top" has to be aware of every lower part and compensate for it

The normal case isn't having A, B, C, and D all overriding E's behavior at different scopes. It's possible and useful to have that capability, but it'd be uncommon to use it. More common with subsumption would be that {B, C, D}—all siblings—override A's behavior in complementary ways for different edge-cases; and then {E, F, G}—all siblings—might override B's behavior, including overriding the inputs it uses to compute its override values for A, but not including the logic it uses to compute that override.

Or, in short: you subsume your "direct reports" by changing their goals and modifying their work products; but you don't micromanage down the org-chart. (Except when micro-managing gets you something.)

Also keep in mind, we're not talking about a tree with one central top-level agent. Robotics systems don't work like that, for several good reasons. It's better viewed upside-down: each one low-level agent is the root of a tree of patches to its behavior, where 1 gets overridden by 2 and 3, 2 gets overridden by {4, 5, 6}, etc.

Notice that a node like 5, that determines how to bias the input into 2, isn't modelling the behavior of 1; it's just trying to directly influence 2 on 2's own terms, with everything below 2 being a black box.

The full system is, then, a forest of agents, where all agents are sourcing shared inputs and are capable of observing any other tree-node's outputs; but there are no agents shared between different trees in the forest, only distinct per-tree copies with similar functions, trying to do the same job while embedded in different local environments.

A good example of subsumption—without any Machine Learning to get in the way—is how a subway system works:

1. Each wheel has a low-level agent in its motor firmware that "wants" to calibrate power output so that the train will continue along the track at a constant speed. It is built as a simple control system: it can see the train's speed, and it will provide more or less power if the train is going faster or slower than its reference. (As well, there's another low-level control system attached to the brakes, that also observe the train's current speed and set-point speed, and will engage the brakes as long as the current speed is much higher than the reference speed.)

2. There's a higher-level agent, still one per motor, responsible for not colliding with things on the track (e.g. other trains.) It can observe when it enters contested sections of track (and, more recently, can read off a front-facing depth-sensor); and will bias the reference-point values for the lower-level wheel-motor and brake systems, so that they "seek to stop" more fervently, the closer the train gets to any obstacle. You can sort of see this like an agent that stands between a gas pedal in a car and the actual signal to the gearbox; but rather than influencing the signal's output for everything, it's an individualized influence for each wheel. (This is important in robotic systems, because hardware is built to varying tolerances—different wheels on the same train can require different amounts of power to generate the same torque!)

3. There's a higher-level agent still—again, still one per motor!—responsible for stopping at stations, which does this by biasing the collision agent to temporarily think there's another train just far-enough away that it must stop where the station is in order to avoid it.

Note how the stopping-at-stations agent doesn't know how the collision-avoidance agent actually does collision avoidance. It doesn't know it's overriding a wheel-power agent further down. It just "delegates" the requirement, by changing the requirements for its subsumed subordinate agent.

And, yes, the stopping-at-stations agent wouldn't get a very good output without some careful fine-tuning of where to produce the illusionary train in the collision-detection agent's input model. That's why ML is used with these systems: to tune inputs like that, in accordance with what parameter the upper-level agent can emit that will best make the lower-level agent do what it wants to accomplish its own goal, without actually needing to understand how the lower-level agent is accomplishing that goal—only the ability to observe how closely the train manages to line up with the station.

If you're wondering why it's all duplicated so much: consider how this system reacts if there's a bug in one motor's sensors that makes it observe the wrong input values. It'll start misbehaving as a result. Certainly, with some kind of online-learning ML in play, the agent above it might realize it needs to apply some sort of gamma curve to ramp the input/output from the agent to get the results it expects. But even without ML, as long as the other wheels are all functioning, they'll fight the misbehaving wheel, and they'll likely win. I.e., even if one wheel-motor is going full tilt because it thinks the train is stopped, the other ones will notice that the train is actually going too fast, and apply their brakes.

It is theorized that this is, in some ways, a model for how synapses work, especially for connections projecting between different parts of your brain (e.g. your neocortex vs. paleocortex.)