Is there any predictability/patterns for neuron/layer activation? If so, would it be reasonable to have a second tiny model that specifically tries to predict activation and preemptively swap those into memory?
For vanilla models, you always use all the weights. That isn't true for mixture-of-experts, though, and in that setting, your approach has merit.
The number of layers, and the amount of time spent in each of them, makes me think any benefit from pre-loading the layer ahead is negligible.
You really need the entire model on device to consider it performant.