That's a good question, and I provided scant few hints towards it in my original post.

There are a few layers in which the order of certain things matter -- basically, any chaotic system arising from the choices made in training neural networks. We oftentimes just randomly choose an answer as that's a "best guess", but in pushing the territory of world records, something more principled is in order.

Weight initialization is one, and what data we show to the network when is another. Each choice at any point influences every single choice that comes after it, so even if it was quantized into discrete decision bins (which it can be for both of those, I believe -- even the weight initialization if you're a lottery ticket hypothesis fan. I cringe in saying that though as that phrase can summon an interesting mix of people.) In that sense, calculating which order of operations/order of values is ideal is I believe an NP-hard problem by definition, and not too much weirdness if we're looking at the discrete case(s) I think.

Maybe solving that up front from a structure perspective is untenable, but if we're able to crack some of the mystery of the solution manifold and turn that into portable priors for architectures like this, then I think that opens the door to connecting things to a more universal, pure mathematical solution. And then that ends up unpacking nicely in other problem domains even if we maybe don't know up front which priors work well for that particular subdomain. If we have some sort of mathematically-connected rule, we can be more sure about it.

That's a loose form of a general workflow I follow, it's a bit more of a crapshoot though where any discrete chaotic processes are involved, unfortunately.

Hope that helps answer some of the question, there's definitely other layers to be had, though. Which means job security for a lot of people for a long time to come, I personally think. :D :))))

There are architectures with layers which can approximate MAXSAT [1] (or rather, an SDP relaxation thereof), but I doubt this is what OP was referring to.

[1]: https://arxiv.org/abs/1905.12149